Raimi war hier

Hm, seems as Debian Etch is converging towards being stable some key magic is to be applied again. Symptom on apt-get update:

W: There are no public key available for the following key IDs:
A70DAF536070D3A1
W: You may want to run apt-get update to correct these problems

Now, as this message explains a proper way of obtaining this key (which seems to be the etch release key) is:

apt-get install debian-archive-keyring/unstable

or when you are using sid (or whysoever):

apt-get install debian-archive-keyring

And that's it. Let's see if I remember that next year :)

PS: I had to move the MySQL database(s) away from the hard disc, too. Turns out that the InnoDB backend writes something every hour and a half or something, even when the application is idle (some kind of check- or savepoint, I guess).

There are still some unaccounted spin-ups that are hard to track down. To get to the bottom of those one requires some tool that can tell when the discs turned on. Together with the respective timestamps a pattern should show. Does anyone have an idea? Perhaps there is a SMART attribute indicating something related to motor movement (I know there is a spin-up time that probably changes a little everytime the disc turns on).

PPS: When I described the hardware of the machine, I forgot to mention something rather annoying: It runs out of random bits. The symptom is a blocking /dev/random device (which produces good random bits) which in turn makes exim block for long amounts of time when delivering mail with SMTP/TLS. Test your entropy:

 cat /proc/sys/kernel/random/entropy_avail

It usually shows values between 3000 and 4000 but on blackbox it frequently goes down to 0. The reason seems to be that the entropy is largely derived from hardware interrupt timings but leaving out sources that might be controlled from the outside (NICs, they say). This excludes most of the IRQ sources on this host (the HDDs are intentionally left idle most of the time).

The workaround is to make the applications use /dev/urandom (which produces pseudo-randomnes). This is actually bad but the only usable solution. I do the switch globally with a symlink which sometimes is reset by a Debian update.

Oh boy. This morning I was struck by this bug - an update of mozilla-thunderbird killed my mail program. Turns out that the debian package thunderbird is no more, it's called icedove now. Why, oh why? Apparently Debian is not allowed to use Mozilla trademarks for their version of Seamonkey, Firefox, Thunderbird (and whatnot).

While this is legally "understandable" it's bad to install different packages now because of some dispute about a logo copyright. So just as a reminder to myself: Thunderbird is icedove, Mozilla is Gnuzilla and Firefox is IceWeasel. I hope those programs keep their files in my home directory in the old locations.

/me shakes his head and goes back to work.

(I would've loved to call this article "Silence of the LAMP" but the machine in question is not primarily a webserver - and I have some issues with the M and P part, but that's yet another article).

As noted below I updated my dialup gateway a couple of days ago and I wanted to follow up with some additional information on how to tweak a machine like that. First some technical details: It's based on the VIA Epia platform with a 1 GHz Nemiah CPU (C3) in a small barebone case with a 200W power unit (ITX, i think). It has 512MB of RAM, 2x160GB HDD as a RAID 1 (mirror). The only PCI slot is filled with an active ISDN card (AVM B1). The board itself is pretty cool for dialup servers, because it has two ethernet NICs (via-rhine) and not much other gimmicks you wouldnt need anyway (audio, several USB ports, even some GPIO stuff which isnt documented, afaik).

Design goals

The machine is to serve my general needs in my home network: give me internet access and protect me from it. It is hooked up to my DSL-modem and ISDN-terminator and works as a masquerading gateway (aka nat) and firewall. It serves my workstation and various other hardware from my museum and whatever friends bring. Mainly DHCP, DNS, Samba shares and a printer, NFS, lpr-printing, internal web server and some special purpose applications as well as shell access when I'm not home.

It is up 24/7 and it is located in my flat. So it has to be as quiet as possible, consume as little power as possible and still be there when I need it.

Hardware choice and preparations

Most consumer hardware is not meant to be running 24/7. Also most people dont want to have a 19" rack in the kichen (I wouldnt mind actually, but i dont have the space). So I went for Epia for its small form factor and because it is meant to be used for appliances like mine. CPU power is not what you would expect from 1Ghz (feels more like 600Mhz on a PIII), but this box is not meant for number-crunching anyway.

For reasons mentioned below the hard disk is a RAID of two mirrored drives. If possible, use drives that are meant to be used in servers. Consumer disks _will_ crash sooner than later (i learned this the hard way). The little extra fee is definitly worth it. Still, a single drive is too dangerous, even monitoring it might not give an advance warning before failure. I have the linux software raid setup on virtually all of my machines now (should I write about it?) and I did play the disk-failure game more than once. Dust is a serious problem for long-running hardware, especially in my flat. The fans dont like it and it makes the interior ugly. The case I use has a grill in the front panel which I vacuum once a year and the little holes on the side are covered with nylon stockings from the inside. Pretty effective as shown here. To reduce case vibration I applied some fragments of sealing strip (about 6, 1cm each) between the frame and the cover. Makes closing the case a little tricky.

Reduce power consumption

Not only for economical reasons is a reduced power consumption a good idea. Energy produces heat which has to be transported out, which requires noise fans (I will leave the philosophical and ethical aspects of energy consumption as an exercise for the reader).

IMHO, there are three main factors that determine energy consumption of a setup like this. First, there is the total amount of energy needed for all related devices. Second is the CPU and third is the hard disk.

When optimizing a setup like this people tend to forget all the little devices we have in our flats that all have a small power converter and constantly need something simple like 5 or 12V DC. Both of which are readily available from the power supply unit of a normal PC. Reducing the number of power units involved reduces the wasted energy because each unit works much less than 100% effective. Also, most PC power units work most effective when they are used with about 80% of their designed output. So adding a device or two is no problem or even a good idea. I hooked up my DSL modem to the machine serving the internet (I dont need the one without the other, when one device fails there is no further reduction of quality-of-life). Another candidate would be the base of the cordless phone that is in the vicinity anyway. The images here show a little adapter that provides 12V DC via a standard plug. The device could convert it to other voltages, too, but I wouldnt trust that cheap little thing too much. I cant even find it in the online shop I bought it anymore (perhaps it turned out to be too dangerous :). But you can tap into the power cables directly or even use some mA from the USB ports (as all those silly gadgets at ThinkGeek do).

Reducing CPU power (consumption)

Most modern CPUs have ways to reduce their power consumption by reducing their speed and by lowering their core voltage. My C3 processor does that with the 'longhaul' kernel module. I use the powernowd to measure CPU utilization and react accordingly. There are more sophisticated approaches but this one is easy to set up and effective without being too intrusive.

Results and current speed are visible in /proc/cpuinfo directly. The idle machine runs at 665Mhz but as soon as something CPU intensive (like compiling a kernel) is done, it goes back to 1Ghz. Keep in mind that such a machine is idle about 99% of the day, it only does some simple tasks when I'm actually at home, awake and using my workstation - and even then it's usually bored.

Reducing hard disc noise

Now, this is probably the most interesting and challenging part of this operation. As stated above the machine is idle most of the day so it is only logical that it should turn off the hard disk motors during that time. This sounds simple because modern drives have built-in logic for that. The problem lies in making sure that nothing happens on the drive when the machine is basically idle. This turns out to be pretty difficult.

First you have to have enough RAM so that everything the running services need end up cached. This is no problem nowadays, the 512MB are plenty (the old box had 96MB and about the same services). The most important thing to do is to mount all filesystems with noatime. It's a little known fact that the filesystem not only records the creation and last-modified timestamps of each file but also when it was last accessed. This way even read-only (and eventually cached) file operations result in traffic on the drive (observing users might notice a short blink of the HDD LED about every 5s). The noatime mount option turns this off.

The next logical thing is to make all running applications stop writing when they are idle. This is mostly syslogd -m 0 to disable the useless "MARK" lines in /var/log/messages. This trick was enough for my old box to send the HDD to sleep most of the day. Unfortunately the new services make it virtually impossible to have silence in /var/log and /var/run - most notably samba which maintains endless databases and logfiles of what is happening on the wire (interesting how much traffic a single redmond-OS host can require to not-use a share and a printer on a samba host). This felt like a dead end.

The solution comes in form of silent, solid state, non-volatile memory: cheap USB 2.0 flash rom sticks. I use one with 1GB - a joke compared with the 1GB my old box had as a HDD. For a simple appliance you could pack the entire OS installation with room for a decade of logfiles on a 2 or 4GB model. This would work even internally with an adapter like this and a compact flash device.

As indicated above, /var/log is not enough to put on that flash memory. The entire /var filesystem on the other hand would be too much because it may contain stuff that grows large: /var/cache/apt for all the downloaded Debian packages and /var/lib/mysql for some local databases. So I put those in /varhdd and bind-mount them to their old place (so that I do not have to reconfigure the services or remember the fact). The fstab contains this:

 /dev/sda1       /var            ext3    noatime,errors=remount-ro 0       0
 /varhdd/cache/apt /var/cache/apt none   bind 0 1
 /varhdd/lib/mysql /var/lib/mysql none   bind 0 1

Of course: USB core and mass storage have to be built into the kernel and the mount points have to exist on the out-sourced /var filesystem.

(Oddly enough this even works with /var/cache/apt being exported via NFS to the other Debian hosts. Seems bind-mounting is different from re-exporting mounted filesystems which works not that easy.)

It may be necessary to find other locations that are written by your applications. I used a crude approach like this (because the lsof manpage would have taken longer to read):

 lsof | sed 's/^\([^ ]*\).* \([^ ]*\)$/\2 - \1/' | sort | uniq | less

To configure the disks to spin down, hdparm has to be set up (/etc/hdparm.conf) for each drive:

/dev/hdc {
        mult_sect_io = 16
        write_cache = on
        dma = on
        spindown_time = 240
}

Read the manpage for the parameter format, it's silly. Those 240 means 20 minutes which works for me. Making it too short results in too many spindown/spinup cycles, making it too long wastes noise and energy.

Conclusion

So far it works. I am yet to measure the actual power consumption of the idle and busy setup which would allow me to calculate how much it actually costs per year. But perhaps I dont wanna know :)

Also, when the disks have to spin up, the i/o-operation can take up to 12s to complete (or even start). So a little patience is required to log into the idle machine.

Finally, the box is not completely silent, yet. There are still three fans: A large one in the power supply unit and two small ones in the case and on top of the CPU. I cannot tell if they are all temperature controlled or if I have to do anything about it. I'll have to experiment with lm-sensors (and the vt1211 module) some more.

It is done. I replaced a machine that was scheduled for demolition for years. The gate went into service around September 1998 to be my ISDN dialup masqurading/NAT gateway to the outside world (with bind8, dhcpd). It was also a vbox-based answering machine and an ermergency X terminal (in the hall).

It was based on SuSE 6.4 or 7.0 and ran with Linux 2.2.21 since about June 2002 and was upgraded for DSL somewhen. It was powered with a 133 Mhz Pentium, 96MB RAM, had a 3-COM and a ne2k NIC (the RTL-8139 committed suicide), an AVM A1 (Fritz) passive ISDN controller (ISA), and an ISA board with 4 additional serial ports (totalling 6 for one Wyse terminal and a mouse). It was rebooted only for kernel upgrades and about once a year for a .25 second power outage the local power company seems to like. The VGA board stopped working around mid-2004 (with the BIOS actually beeping on reboot because it couldnt find it anymore).

Ever since about March 2004 I was waiting for the 1 GB Quantum Fireball HDD to crash and force me to move to other hardware. The disc had an automatic spin-down and was powered off most of the day, which seems to have increased the life expectancy. However, back in the day this Fireball was known for being easily damageable.

So for 2 or 3 years I was literally waiting for the machine to fail. I bought a replacement already (blackbox based on a VIA Nemiah barebone) but had to replace the Maxtor disc already, losing all data without prior waring (smartmontools where running).

Since yesterday blackbox is serving NAT and other stuff (DNS, dhcpd, printer, NFS, Samba for the guests, ISDN (AVM B1 active card) and whatnot alone. I had to switch so that I could have VPN access to some place that uses un-tunnelable protocols.

Lessons learned: (1) There's life in the old dog yet. (2) Never touch a running system (3) ...unless you have to.

I just _know_ the prematurely switched-off hardware will seek revenge. gate was still good and it knows it. I'm so afraid. I will post funeral photos once I get myself a mask. There's enough dust in and around the case to feed several billion mites for a decade.

Funny. Due to missing exposure and a "comment" feature there is not much feedback I get for this blog. The entry that triggered most of it yet, is this one about an unsolvable Oracle/PHP problem.

So for the both of you that had the same problem, the archives and the search index: Should your OCI8 client program hang for no reason with the 10.2.0.1 client: downgrade. I solved it by downgrading to 10.1.0.4 but this known bug is supposedly fixed in 10.2.0.2 but I did not test it. Thanks for the feedback, guys.

Call off the rescue team, I made it out alive. Question: How do you make Oracle 10g R2 (Enterprise Database) run on Debian GNU/Linux on AMD64, a system that is not mentioned in the "Certify" certification matrix. Short answer: You dont.

Long answer: You make it believe it runs on a certified host. As mentioned below, the original approach was to just try run it on Debian install missing old libraries until a fixpoint is reached. Since the "leading enterise database" heavily relies on an unhealthy mix of 32- and 64-bit libraries, binaries and java interpreters this turned out to be hopeless (at least on a pure 64-bit system Debian still is). Going 32-bit didnt sound good with all the hassle I was about to start.

I chose SLES9 (SuSE Linux Enterprise Server 9) as the sandbox system for the Oracle because I used to use SuSE once uppon a time when Linux was all new to me (so no, I dont like Red Hat either). Boy, am I glad I switched to a sane distro years ago. I wouldnt stand SuSE anymore. Unfortenately my fine Opterons are not good enough for 64-bit guests in a vmware. So I went for Qemu to emulate a 64-Bit machine on my 64-Bit machine and (painfully) installed SLES9 into it. This took almost a day with Qemu being a little slilly with its command line arguments and SuSE being a major PITA doing anything.

Running the oracle's OUI inside Qemu resulted in a segfault, so I copied the root filesystem out of the emulated machine to /sles9 getting a 64-Bit chroot jail to run SuSE. Originally I wanted to do this after Oracle ran. Both the host OS and the guest have the oracle users/groups in common, this simplyfies things. Now, inside this chroot (or rather schroot) jail I just ran the installer without any problems. Really, without _any_ problems, which is totally new to me. Of course, I had to (yet again) guess which programs to invoke in my init scripts to automatically start the listener, the db itself, the enterprise manager and the isqlplus thingie. But I expected that.

So what do I gain (after bind-mounting /proc any other imported filesystems into the jail)? My custom 64-bit kernel running my dear Debian host and a SLES9 jail which doesnt even know it's not running as the primary OS. Cool, best of all worlds.

Now, go do that with your toy OS :)

I dont get it, really. I was about to install Oracle 10g (10.2.0.1) on a Debian amd64/x86_64 system.. I was yet again so foolish to assume that this would just work. Sure, the compatibility list doesnt mention Debian, but it never does and it usually works anyway. The restriction to a specific distribution is both insane and useless. But this time they're really into it.

Once i thought Oracles Universal Installer (OUI) was a neat idea - it's written in Java and would work cross-platform. They screwed up by shipping their own ancient/broken dynamically linked JRE without shipping the ancient libraries it is linked against. Oh wait, it's actually 1.4.2. But hey, it's the 32-bit JRE - oh boy, am I lucky I downloaded 700MB of x86_64-specific software. Damit, just let me use my own JRE and we both can be fine! It's the entire point of platform independent software to let the user supply the platform and stop wild-guessing what I might have.

Sidenode: Some years ago I tried to install Oracle 9 on a windoze host - the JRE failed because the then-shipped ancient JRE couldnt make sense of my then-new P4 CPU. Deleting the b0rked symantec JIT dll file made it work then. I guess this was a warning shot.

Now, for a brief moment I considered hacking the installer so that it would use my own JRE but then I found this Debian- Ubuntu-related bug that instantly crushed all my hopes.

My current plans involve secretly torturing Larry E. by installing both him and the 10g in my 32-bit chroot jail. I somehow have the feeling the DB kernel wouldnt take advantage of any 64-bit features anyway :)

If I dont report back on tuesday, please send a rescue team.

My family of hardware (running Linux) got extended by an (old) Sony VAIO PCG-SR21K (or PCG-3316 as printed on the bottom) called bob.

Installing was a little tricky because the Debian (and Ubuntu) installers dont come with the kernel modules required to access the externel PCMCIA CD-ROM drive. So I booted from the CD and used a Memory Stick (tm) to access the netinst image. As a side effect I could insert the PCMCIA network card for the netinst (there is only one PCMCIA slot).

Bob himself is cute. The hard disc is surprisingly fast and the display at 1024x768 is terrific for its size. CPU performance is within expected range and the S3 savage chip accellerates enough for full-screen video (I just love mplayer).

Now for the stuff that really matters: With this hint i got Speedstep working: speedstep_smi.smi_cmd=0x82. This project has a kernel driver and command line tools (spicctrl) that work with the Fn key and Jogdial. Making the dial generate mouse events works already and the Fn keys trigger their respective events. What I am lacking is something nice (and not X-related) to poll for these events and perform suspend operations. This wouldnt be too hard since acpitool -s sends the machine to sleep just fine. But when it wakes up, the TFT backlight is not turned on. So this is what I'm lacking.

PS: The built-in speakers suck and the thing gets really warm. Seems the reported 47 to 70 degree celsius are completely transported to my lap.

Some co-workers and me (I?) use KOrganizer to access the company ical calenders (about a dozen) via webdav. We used to use Mozilla Sunbird but that program was way too slow, too old and too unstable.

I'm not really a KDE fan but that Korg-anizer is a usable tool. Rather quick and mostly functional. Unfortunately we got hit by this bug, which makes the program leak tons of memory (several hundred MB within a week, we got one killed with 1.7GB). For that reason we have the root user at the Linux workstations with this cron job:

0 1 * * * killall korgac korganizer

Oh, and PS: It would be nice if the "Remember password" checkbox was functional. And also it shouldn't screw up its tempfiles several times a day. And copying the configuration (~/.kde/share/config/kresources/calendar/stdrc) should transport the calendar colors as well.

This is a general warning: Do not use megaraid SATA RAID controllers (LSI Logic / Symbios Logic PowerEdge Expandable RAID Controller 4 (rev 01), also rebranded and sold by major manufacturers).

We had a machine with both Linux 2.4 and 2.6 running (various versions), Seagate and WDC disks, tried to tune software and BIOS parameters. Nothing helped. This is what we took out of order:

megaraid cmm: 2.20.2.5 (Release Date: Fri Jan 21 00:01:03 EST 2005)
megaraid: 2.20.4.5 (Release Date: Thu Feb 03 12:27:22 EST 2005)
megaraid: fw version:[713G] bios version:[G117]
scsi0 : LSI Logic MegaRAID driver

This controller does not deliver more than 6MB/s when reading from a mirrored RAID. Sustained write performance is something like 4MB/s, but that's not the worst part: You can write very quickly to it until the on-board cache is full (64MB). While this cache is being flushed the host system comes to a complete halt since every disk access is suspended. The system becomes unusable.

We solved the problem by throwing out the controller replacing it with a Linux software raid. Suddenly the disks read and write happily at about 40MB/s. The fun part about this fact is that both the on-board SATA chip and the chips on the LSI controller are Sil 3114. It's the same thing, LSI just screwed it up.

We got rid of the controller several months back. If you are in the unlucky position of owning this device you might try flashing a new controller BIOS (which will be fun if your server (like ours) doesnt have a removable drive and doesnt run a Redmond-based OS) -- but this is just a straw.

The reason why I blog this now is that some of our customers have the same problem and others even buy new machines with this controller (Please Michael, stop selling them). When i started having performance problems i couldnt find negative reports on the net so this is my attempt to raise some noise.

I'm still getting used to the alternatives system of Debian which is used to allow parallel installation of equivalent packages. Some packages ship this way, some (as gcc) need this being done by hand:

update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-3.3 1
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-3.4 2
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.0 3

The last argument is the priority, the highest priority wins and is the default. Here is how to change your selection:

# update-alternatives --config gcc

There are 3 alternatives which provide `gcc'.

  Selection    Alternative
_______________________________________________
      1        /usr/bin/gcc-3.3
      2        /usr/bin/gcc-3.4
*+    3        /usr/bin/gcc-4.0

Press enter to keep the default[*], or type selection number: 3
Using `/usr/bin/gcc-4.0' to provide `gcc'.

And there you go, have different gcc installed at once and have a global default. Please note: This is for site-defaults, not user preferences.

Repeat

wget http://ftp-master.debian.org/ziyi_key_2006.asc -O - | apt-key add -

until the message gpg: no ultimately trusted keys found goes away and only OK is shown (2 times for me). Verify with apt-key list which shows

pub   1024D/2D230C5F 2006-01-03 [expires: 2007-02-07]
uid   Debian Archive Automatic Signing Key (2006) <ftpmaster@debian.org>

Repeat on all hosts.

Here is what I did to let my (command line) PHP connect an Oracle database. First I downloaded the Instant Client and converted it to Debian (everything done as root):

apt-get install alien
alien oracle-instantclient-basic-10.2.0.1-1.i386.rpm
alien oracle-instantclient-devel-10.2.0.1-1.i386.rpm
alien oracle-instantclient-sqlplus-10.2.0.1-1.i386.rpm

Then I installed the packages and repaired all the file ownerships/permissions since they were unusable (might be solvable with alien):

dpkg -i oracle-instantclient-basic_10.2.0.1-2_i386.deb
dpkg -i oracle-instantclient-devel_10.2.0.1-2_i386.deb
dpkg -i oracle-instantclient-sqlplus_10.2.0.1-2_i386.deb
chown -R root.root /usr/lib/oracle /usr/include/oracle /usr/share/oracle
chmod -R go+r /usr/lib/oracle /usr/include/oracle /usr/share/oracle
find /usr/lib/oracle /usr/include/oracle /usr/share/oracle -type d -exec chmod go+rx {} \;

Everything is installed in those three directories (plus the Debian specific doc fragments). Now the libraries are to be made known to everyone. Edit /etc/ld.so.conf and add:

/usr/lib/oracle/10.2.0.1/client/lib

Now update the linker cache:

ldconfig

The PHP extension itself is installed with PHP Pear/PECL:

apt-get install php4-pear
pear install oci8

When prompted, type instantclient,/usr/lib/oracle/10.2.0.1/client/lib. After the extension is installed, activate it by editing /etc/php4/cli/php.ini and adding

extension=oci8.so

This will install the most current API version, so even the stuff marked as being PHP 5 works in PHP 4. This should work with CGI/Apache-Versions of PHP, just make sure ldconfig has been run before the server starts.

As noted earlier, I am still waiting for 2.6.15 to be released. In the meantime my machine crashed several times and the root filesystem got b0rked. Not fatally, just enough to force a fsck at every reboot which did not finish because the fs contains errors. This makes me wonder how I actually managed to repair a broken root fs up to now. As I learned the hard way, you must not fsck a mounted fs (even read-only mounts make it be destroyed in a snap). But since the fsck program is on the root fs and since you cannot umount / in single-user mode - how am I supposed to repair the root fs? This time i bootet a Debian netinst boot cd in expert mode (what else). The tricky part is to bring up software raid since the autorun (mdrun) doesnt find existing raids (for reasons beyond my comprehension). The trick is to mdadm -Ac partitions -m 0 /dev/md/0 which scans automatically the partitions from /proc/partitions and assembles everything belonging to minor number 0 into /dev/md/0 (1 for /dev/md/1). Then I could fsck.ext3 /dev/md/0, holding my breath everytime I had to type 'y'. But it worked out and my root fs is now okay. ...or should I go for 2.6.14.5 for instant relieve?