[Discuss] I think my server is running out of "something"

[david at thekramers.net (David Kramer)]

On 03/18/2012 05:14 PM, Tom Metro wrote:
> David Kramer wrote:
>> Home-built server/firewall/mail server/web server/MythTV back end/makes
>> coffee.
> 
> Others covered this...I'll only add that a router appliance would
> provide a fairly cheap "upgrade", hardly add any additional power usage,
> and improve your security, while offloading some of your server workload.

I feel very strongly that this is not related to my problem, which
presents itself at boot time.  I do actually have a WRT54G that I'm not
using because it would lock up when it saw too much load, but maybe it's
a software thing, and it would be fine with different software.  I'll
bring it to the installfest and if anyone has time to help me with it,
then that's great, otherwise it will have to wait until I have MUCH more
free time.

If there is any real-world benefit to this at all, it will be a slight
drop in load by having the router block access to ports I don't have
open instead of my server having to handle them.

>> I decided to add another 1TB drive...the server won't boot.
> 
> What exactly do you see when it fails to boot?

"DISK BOOT FAILURE, INSERT SYSTEM DISK AND PRESS ENTER"

>> It's a Silverstone Olympia OP640  650W, which should be plenty.
>> It's possible that just the SATA rail is maxed out.
> 
> I concur with the others that this sounds like a power supply issue,
> though 6 hard drives should not be excessive for a 650W supply. (I have
> as many or more drives on my MythTV server with something like a 450W
> supply.) But if the supply is optimized for running high powered
> graphics cards it may in fact be running out of current on the 12V line
> used by the drives.
> 
> 
>> Maybe I should try putting the drive in an external sata case for 
>> power...
> 
> That wold be a good test and possibly a permanent work around.

I did one better.  I borrowed a whole computer from a friend that has a
sata connector (yes, just one.  Very strange) on the power supply, and
powered my new hard drive from that, and ran the sata cable from it to
my server.  I got identical results: with the DVDRW hooked up, I got the
same disk boot failure message, and with it removed, it would boot up.
Sounds like the power supply is not the issue.

But now that I got to playing with it some, I noticed that every few
seconds or a couple of times a minute the system would seem to freeze
then continue.  "top -c" showed very high idle time and low utilization,
and I saw nothing in /var/log/messages.  During the freezes the mouse
would not move and the clock did not change (I leave seconds turned on
for EXACTLY this reason).  When it would come out of a freeze, the mouse
would often be a bit jerky for a few seconds.

However, I was able to use gparted to partition the drive.  The fact
that partitioning worked leads me to believe that the "freezes" might be
a straight UI thing.  I would consider debugging this angle by running a
script that writes the time to a file in an endless loop (running in a
text mode tty, not the GUI).

After partitioning the drive, I updated /etc/fstab to mount it and
created the mount directory and made it 777.  When I rebooted, I got a
message "/DATA4 NOT READY...S TO SKIP OR M TO MANUALLY CONFIGURE" or
something close to that.  I issued the mount command, and it mounted
fine.  Very strange.  I redid this test and got the same results.

The /var/log/messages showed no problems with the drive (/dev/sde)
during boot:
sd 7:0:0:0: [sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
sd 7:0:0:0: [sde] Write Protect is off
sd 7:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't
support DPO or FUA
 sde:
sd 6:0:0:0: [sdd] Write Protect is off
sd 6:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't
support DPO or FUA
 sdd: sdc1
 sdd1
sd 5:0:0:0: [sdc] Attached SCSI disk
sd 6:0:0:0: [sdd] Attached SCSI disk
 sde1
sd 7:0:0:0: [sde] Attached SCSI disk

When I manually mounted it, /var/log/messages showed
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sde1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.

Why it mounts on demand but not automatically is baffling.

The current state is that I unhooked that drive and hooked by DVDRW up
again.  Now the system is running fine with no periodic freezes.  Though
I admit I forgot to comment out the new line in /etc/fstab and had to
edit it and ^M.


Unless someone else has ideas, here are my next steps:
- Try a different sata cable (LONG shot.  Brand new cable).

- See if there's anything else I can disable in the BIOS, like parallel
port, and remove any nonessential USB devices to see if my original
assumption of running out of IRQ/DMA is the problem.

- Put the new drive in my friend's machine and see if I can install
Ubuntu on it and boot off of it, proving the drive works fine on its own.

Any other ideas?