Raimi war hier

PS: I had to move the MySQL database(s) away from the hard disc, too. Turns out that the InnoDB backend writes something every hour and a half or something, even when the application is idle (some kind of check- or savepoint, I guess).

There are still some unaccounted spin-ups that are hard to track down. To get to the bottom of those one requires some tool that can tell when the discs turned on. Together with the respective timestamps a pattern should show. Does anyone have an idea? Perhaps there is a SMART attribute indicating something related to motor movement (I know there is a spin-up time that probably changes a little everytime the disc turns on).

PPS: When I described the hardware of the machine, I forgot to mention something rather annoying: It runs out of random bits. The symptom is a blocking /dev/random device (which produces good random bits) which in turn makes exim block for long amounts of time when delivering mail with SMTP/TLS. Test your entropy:

 cat /proc/sys/kernel/random/entropy_avail

It usually shows values between 3000 and 4000 but on blackbox it frequently goes down to 0. The reason seems to be that the entropy is largely derived from hardware interrupt timings but leaving out sources that might be controlled from the outside (NICs, they say). This excludes most of the IRQ sources on this host (the HDDs are intentionally left idle most of the time).

The workaround is to make the applications use /dev/urandom (which produces pseudo-randomnes). This is actually bad but the only usable solution. I do the switch globally with a symlink which sometimes is reset by a Debian update.

Oh boy. This morning I was struck by this bug - an update of mozilla-thunderbird killed my mail program. Turns out that the debian package thunderbird is no more, it's called icedove now. Why, oh why? Apparently Debian is not allowed to use Mozilla trademarks for their version of Seamonkey, Firefox, Thunderbird (and whatnot).

While this is legally "understandable" it's bad to install different packages now because of some dispute about a logo copyright. So just as a reminder to myself: Thunderbird is icedove, Mozilla is Gnuzilla and Firefox is IceWeasel. I hope those programs keep their files in my home directory in the old locations.

/me shakes his head and goes back to work.

(I would've loved to call this article "Silence of the LAMP" but the machine in question is not primarily a webserver - and I have some issues with the M and P part, but that's yet another article).

As noted below I updated my dialup gateway a couple of days ago and I wanted to follow up with some additional information on how to tweak a machine like that. First some technical details: It's based on the VIA Epia platform with a 1 GHz Nemiah CPU (C3) in a small barebone case with a 200W power unit (ITX, i think). It has 512MB of RAM, 2x160GB HDD as a RAID 1 (mirror). The only PCI slot is filled with an active ISDN card (AVM B1). The board itself is pretty cool for dialup servers, because it has two ethernet NICs (via-rhine) and not much other gimmicks you wouldnt need anyway (audio, several USB ports, even some GPIO stuff which isnt documented, afaik).

Design goals

The machine is to serve my general needs in my home network: give me internet access and protect me from it. It is hooked up to my DSL-modem and ISDN-terminator and works as a masquerading gateway (aka nat) and firewall. It serves my workstation and various other hardware from my museum and whatever friends bring. Mainly DHCP, DNS, Samba shares and a printer, NFS, lpr-printing, internal web server and some special purpose applications as well as shell access when I'm not home.

It is up 24/7 and it is located in my flat. So it has to be as quiet as possible, consume as little power as possible and still be there when I need it.

Hardware choice and preparations

Most consumer hardware is not meant to be running 24/7. Also most people dont want to have a 19" rack in the kichen (I wouldnt mind actually, but i dont have the space). So I went for Epia for its small form factor and because it is meant to be used for appliances like mine. CPU power is not what you would expect from 1Ghz (feels more like 600Mhz on a PIII), but this box is not meant for number-crunching anyway.

For reasons mentioned below the hard disk is a RAID of two mirrored drives. If possible, use drives that are meant to be used in servers. Consumer disks _will_ crash sooner than later (i learned this the hard way). The little extra fee is definitly worth it. Still, a single drive is too dangerous, even monitoring it might not give an advance warning before failure. I have the linux software raid setup on virtually all of my machines now (should I write about it?) and I did play the disk-failure game more than once. Dust is a serious problem for long-running hardware, especially in my flat. The fans dont like it and it makes the interior ugly. The case I use has a grill in the front panel which I vacuum once a year and the little holes on the side are covered with nylon stockings from the inside. Pretty effective as shown here. To reduce case vibration I applied some fragments of sealing strip (about 6, 1cm each) between the frame and the cover. Makes closing the case a little tricky.

Reduce power consumption

Not only for economical reasons is a reduced power consumption a good idea. Energy produces heat which has to be transported out, which requires noise fans (I will leave the philosophical and ethical aspects of energy consumption as an exercise for the reader).

IMHO, there are three main factors that determine energy consumption of a setup like this. First, there is the total amount of energy needed for all related devices. Second is the CPU and third is the hard disk.

When optimizing a setup like this people tend to forget all the little devices we have in our flats that all have a small power converter and constantly need something simple like 5 or 12V DC. Both of which are readily available from the power supply unit of a normal PC. Reducing the number of power units involved reduces the wasted energy because each unit works much less than 100% effective. Also, most PC power units work most effective when they are used with about 80% of their designed output. So adding a device or two is no problem or even a good idea. I hooked up my DSL modem to the machine serving the internet (I dont need the one without the other, when one device fails there is no further reduction of quality-of-life). Another candidate would be the base of the cordless phone that is in the vicinity anyway. The images here show a little adapter that provides 12V DC via a standard plug. The device could convert it to other voltages, too, but I wouldnt trust that cheap little thing too much. I cant even find it in the online shop I bought it anymore (perhaps it turned out to be too dangerous :). But you can tap into the power cables directly or even use some mA from the USB ports (as all those silly gadgets at ThinkGeek do).

Reducing CPU power (consumption)

Most modern CPUs have ways to reduce their power consumption by reducing their speed and by lowering their core voltage. My C3 processor does that with the 'longhaul' kernel module. I use the powernowd to measure CPU utilization and react accordingly. There are more sophisticated approaches but this one is easy to set up and effective without being too intrusive.

Results and current speed are visible in /proc/cpuinfo directly. The idle machine runs at 665Mhz but as soon as something CPU intensive (like compiling a kernel) is done, it goes back to 1Ghz. Keep in mind that such a machine is idle about 99% of the day, it only does some simple tasks when I'm actually at home, awake and using my workstation - and even then it's usually bored.

Reducing hard disc noise

Now, this is probably the most interesting and challenging part of this operation. As stated above the machine is idle most of the day so it is only logical that it should turn off the hard disk motors during that time. This sounds simple because modern drives have built-in logic for that. The problem lies in making sure that nothing happens on the drive when the machine is basically idle. This turns out to be pretty difficult.

First you have to have enough RAM so that everything the running services need end up cached. This is no problem nowadays, the 512MB are plenty (the old box had 96MB and about the same services). The most important thing to do is to mount all filesystems with noatime. It's a little known fact that the filesystem not only records the creation and last-modified timestamps of each file but also when it was last accessed. This way even read-only (and eventually cached) file operations result in traffic on the drive (observing users might notice a short blink of the HDD LED about every 5s). The noatime mount option turns this off.

The next logical thing is to make all running applications stop writing when they are idle. This is mostly syslogd -m 0 to disable the useless "MARK" lines in /var/log/messages. This trick was enough for my old box to send the HDD to sleep most of the day. Unfortunately the new services make it virtually impossible to have silence in /var/log and /var/run - most notably samba which maintains endless databases and logfiles of what is happening on the wire (interesting how much traffic a single redmond-OS host can require to not-use a share and a printer on a samba host). This felt like a dead end.

The solution comes in form of silent, solid state, non-volatile memory: cheap USB 2.0 flash rom sticks. I use one with 1GB - a joke compared with the 1GB my old box had as a HDD. For a simple appliance you could pack the entire OS installation with room for a decade of logfiles on a 2 or 4GB model. This would work even internally with an adapter like this and a compact flash device.

As indicated above, /var/log is not enough to put on that flash memory. The entire /var filesystem on the other hand would be too much because it may contain stuff that grows large: /var/cache/apt for all the downloaded Debian packages and /var/lib/mysql for some local databases. So I put those in /varhdd and bind-mount them to their old place (so that I do not have to reconfigure the services or remember the fact). The fstab contains this:

 /dev/sda1       /var            ext3    noatime,errors=remount-ro 0       0
 /varhdd/cache/apt /var/cache/apt none   bind 0 1
 /varhdd/lib/mysql /var/lib/mysql none   bind 0 1

Of course: USB core and mass storage have to be built into the kernel and the mount points have to exist on the out-sourced /var filesystem.

(Oddly enough this even works with /var/cache/apt being exported via NFS to the other Debian hosts. Seems bind-mounting is different from re-exporting mounted filesystems which works not that easy.)

It may be necessary to find other locations that are written by your applications. I used a crude approach like this (because the lsof manpage would have taken longer to read):

 lsof | sed 's/^\([^ ]*\).* \([^ ]*\)$/\2 - \1/' | sort | uniq | less

To configure the disks to spin down, hdparm has to be set up (/etc/hdparm.conf) for each drive:

/dev/hdc {
        mult_sect_io = 16
        write_cache = on
        dma = on
        spindown_time = 240
}

Read the manpage for the parameter format, it's silly. Those 240 means 20 minutes which works for me. Making it too short results in too many spindown/spinup cycles, making it too long wastes noise and energy.

Conclusion

So far it works. I am yet to measure the actual power consumption of the idle and busy setup which would allow me to calculate how much it actually costs per year. But perhaps I dont wanna know :)

Also, when the disks have to spin up, the i/o-operation can take up to 12s to complete (or even start). So a little patience is required to log into the idle machine.

Finally, the box is not completely silent, yet. There are still three fans: A large one in the power supply unit and two small ones in the case and on top of the CPU. I cannot tell if they are all temperature controlled or if I have to do anything about it. I'll have to experiment with lm-sensors (and the vt1211 module) some more.