This was originally going to be a very long post, but I kept putting this off ... and now I just feel that something needs to be said.
The story starts with waking up on February 15th, to find zen was dead. It had self updated overnight, and now it was unbootable, and the start repair couldn't get me back. Apparently, the problem had started long ago with all the previous times where Windows 7 would lock up...usually under intense disk activity...and the afterwards, the intel matrix raid would require re-initialization of my 1.5TB RAID 1 array.
Apparently, it was slowly corrupting my drive....because trying to restore from WindowsImageBackup was also a failure. Since this happened the day before Gallifrey One, I had to wait until I got back to do some more serious attempts to recovery, during which I ordered a full copy of Windows 7 Professional, hoping that a repair install might be an option. It isn't because the repair option can only be invoked inside a running Windows 7 system .... in need of repair. Not by booting the disk. ARGH!
At least I should have the data in BackupPC to restore from.....though hopefully before the bit rot of its ext4 filesystem makes it go away. Plus I had hoped to get some configuration going where I could mount the RR62x RAID 5 array, and get at the Oops!Backup store.
So, the plan now was to wait until Ubuntu 12.04LTS to land and then maybe some configuration of running Windows 7 in VirtualBox and recovering into that, etc.
When I had gotten Zen (HPE-450t), I couldn't resist and opened it up to see what it had inside, etc. And, one of the things I noted was that there were two available PCIe x1 slots.
Later during the initial setup, I found myself running out of physical USB 2.0 ports on the back. I did have a 7 port USB hub, but there's so many things out there that just don't like the extra layer of a hub. And, I had a lot of this things. While, I did later get a 24-port USB hub to help get everything connected to Zen. But, I felt that I needed to eventually get something in additional real USB 2.0 ports.
And, eventually I settled on wanting to get a StarTech 7 Port PCIe LP USB 2.0 Card (PEXUSB7LP). It actually only has 6 ports on the back, but it was the highest density PCIe x1 card I could find (there's an internal header, but its no use for me). I bought this back on Feb 3rd using Amazon Prime, of course.
Later as I was thinking of making my local backups go faster, I thought that I would grab one of the extra SATA RAID adapters (SIL3132 based)...that had come with the TR5M or its Rosewill clone...I didn't use either, because I had purchased on before I knew I was getting these enclosures for orac. And, the TR2UT that I had gotten initially didn't have a card....
Put that in Zen, and change how the TR2UT is connected to Zen now. Currently its using its built-in RAID1 ability and connected via USB2.0. I had tried both kinds of Windows 7 backups, but the file backups were taking longer than 24 hours to do the daily differentials...so I stopped doing that. But, the System Image Backup had saved me once before, so even though its a pain...I do that weekly...or try to.
I makes Zen pretty slow...the VSS really hurts things I guess, and it can take close to a day to run. Though it had been getting shorter, though for no reason that I know of. I used to start it at 11pm on Tuesdays, in the hope that it wouldn't impact me too badly come Wednesday night (I tend to be out the early part, so it would help). But, then it started taking less time. Perhaps there's been a Windows update to it. So, now I start it at 10am on Wednesday...usually well after I get my morning stuff done and off to work...and hope that in finishes before I return from my Wednesday evening activities....
Or come home to find out that it failed for some unknown reason and try again.
So anyways...on February 13th, I evidently bought a 2m eSATA cable from Amazon Prime for that project.
Well, I was getting really annoyed with it complaining and kicking out the replaced disk in LHAVEN. So, I shutdown the system and took out the disk, figuring it should just continue to run fine degraded until I get around to replacing it.
Well, it wouldn't power on after that. I went through everything, no joy. It had done this a while back when I tried adding something to it. It could just be the power supply has flaked out, but I don't have a tester or multimeter handy...so I can't really test it. Could buy a new PS...and I may do that.
But, I kind of suspected the drive issues weren't actually the drive but possibly some deeper hardware problem. So, I had been planning to replace LHAVEN at some point. It has done quite well, being circa 2002. Started out as a 64MB Duron 800MHz machine, eventually peaked at 2GB Athlon XP 3000+. It had replaced a Cyrix PR233 box that had gone up in smoke during a hot summer day in 2002, when the transformer outside blew....the computer survived the brownout and then blackout, but the fans didn't spin back up when power returned...so it burned itself up. Later I found one problem with this new system. It wouldn't resume after losing power. No BIOS setting to alter this behavior and I did try to see if there was alternate BIOS updates for it. It was kind of a painful machine to manage, because for some time kernels didn't have built-in support for all the SIS chipset stuff in it....so an update would roll out, and NIC was usually one of the things to definitely go missing. Which made for fun to rebuild a custom kernel upgrade for it. Eventually it got stable.
But, needing essentials like DNS, DHCP to be available after an extended outage...I moved these to another server (originally an old Pentium 75, which has evolved into what is known as 'box' today). For the longest time it was RedHat 7.2, and then RedHat 7.3 when fedora legacy switched to only supporting 7.3 and 9 releases. It continued after fedoralegacy stopped supporting it, I was building some of the packages for it by hand...like bind (in response to the Kaminsky exploit). There were parts of my network that wouldn't function without LHAVEN...like an old Windows 2000 box that was barely functioning...(old Gumby).
I nearly lost it during the Icepocalyse....but I tracked down motherboard replacement for it from geeks.com. It was a slighty newer mobo, but equivalent chipset. It did have some things the old mobo didn't, like USB 2.0 support (I didn't use USB and still don't), support for 2GB of RAM instead of 1GB...so I upped it for better BOINC'ng. And, faster CPUs....so I upped from 2200 to 3000.
At one time, I had turned it off to do some upgrades...which didn't pan out. Tried to slap a gigabit card into it...didn't work. Wouldn't get along with the on board stuff, and the BIOS didn't have ways to get things out of the way enough. It also never fixed the resume after power loss issue. But, during this process it failed to power on....but after a couple days, it came back and I decided to upgrade its UPS and hopefully it would make one last outage. Well, there were several others since then where it came back afterwards. But, looks like this is the last time for it.
Pages: 1· 2
.... kind of, though more not.
Three weeks ago today, Zen locked up. And, when it came back...it wasn't all there. It was bad....stuff wouldn't run, missing or corrupt system dlls, etc. Well, I did buy Acronis True Image Home 2011 the weekend before, and I had got nonstop backups going for a few days...and it seemed to show that it had replaced the old backup settings (image back only). And, trying to go into "Backup and Restore" launched Acronis True Image Home....
Guess I will restore the system to a previous time earlier that day. Seems the last time was about 6 hours back (and there were other weird gaps) later I read that non-stop isn't really non-stop...it'll pause if the machine is 'too busy'. Problem is it thinks BOINC is the machine being 'too busy'....didn't figure this out until later.
But, I went for it. After doing the restore, it rebooted...and...poof, I have nothing. Well, it went into HP's recovery partition instead. And, the only option enabled there is to restore the system to factory. After some cursing, I figured I'd go ahead, reinstall Acronis True Image Home and try again with an earlier time. While, that was happening, I had gotten onto another computer to do some surfing to see if there was some alternative way to invoke the restore, having not used the create restore cd option yet.
I found that I could download an iso from acronis, that was the same as what the software would've created and boot with that. So, instead of completing the initial setup screens of the freshing restored to factory...I booted from that. And, then tried looking for an earlier restore time.
Well, that interface is pretty bad. The worst part is that it sorts non-stop backup times alphabetically....so 1:59 is followed by 10:00, and then 11 and 12, before 2:00 shows....and AM & PM, those are at the end of the string...so PM times are interleaved into the AM times. Well, I remembered what time I had restored to previously, so I had a starting time to search for and try to guess backwards to a suitable restore time. There was a huge gap, now it was a busy 'vacation' day. I had a doctor appoint in the morning and then an MRI in the afternoon. The first restore attempt had been to a time shortly after returning from the MRI. The next backup times were for a period after I had returned from the doctor appointment. Might be okay....I would probably lose one Quicken transaction, but I could easily put that back (entering that I had paid co-pay)...I had read emails then as well as after the MRI, but I didn't recall anything too important that I had to get back having come in since my morning email check.
So, off it went....oh the first restore attempt took hours and I had gone to bed hoping to find Zen back in the morning. Now the next day, I was running a bit late for work...but I got it going, with it reformatting and updating partition settings first and I hoped to come home from work and be up and running again.
Well, that's not what I found. I was back at the HP Recovery Partition screen. So, I rebooted from the Acronis recovery disk, and this time I hit the search for backups option to see if it might find anything else. I figured it was worth a shot, while I mulled over the idea of starting over from scratch....at least I had the stuff in backuppc (sort of). Well, it did find something else...it found some image backups...odd. I had played around the option in Acronis True Image Home, but thought I had opted to go with non-stop backups and a fresh start with the external drive.
But, it seemed to be valid images and it was from the night before the crash. So, I used that to restore. I only made one mistake, where I marked the C: drive as active instead of the hidden system partition. Booting a GParted Live CD, I switched things back and that got me going again. Now while I was in the GParted Live CD, I noticed that it saw two internal drives...instead of the Intel Matrix RaidVol. I suppose that means the Intel ICH8R Matrix Raid is actually another FakeRAID, and after some searching around I found it to be true.
Hmmm, wonder if I shouldn't have bothered ordering the system as RAID1 then? Years ago, I had built a PC using FakeRAID...and it was kind of a mess after the motherboard failed (around the time where lots of motherboard makers were having trouble with counterfeit capacitors.) Yet, I'm considering building a machine from scratch again....
Wonder if it being FakeRAID is why after crashes it messes up the RAID, and has to initialize again (was scary seeing that the first time)...and then verify and fix errors.... wonder if this is going to happen every time Windows 7 fails to shutdown safely?
Oh well, try the boot....and success its all working again. I then look at the backup drive to see what I had restored from. Well, before I setup Acronis True Image Home 2011, I had been using Microsoft's image backup option, and scheduled it to run every Tuesday night. And, it usually runs on through to Wednesday night. Though it did seem to not take as much time lately....guess Zen also getting hit by backuppc at night was slowing the image backup down. I had looked at blocking out that night in backuppc, And, at this particular time...there was no hit from backuppc, because I had taken I was in the process of building a new array (I plan to write about that later) I was between backups.
Guess I'll leave Microsoft's image backup going...since it worked. Though it has one annoying negative. It removes the previous backup before creating the new backup, and if something goes wrong...no backups.
Meanwhile, I couldn't get non-stop backup to go again...probably a mismatch in state or something?
I then dig through the non-stop backup and restore a few files that had changed since the image backup.
And, it will be....but first I decided that I should get at the clutter of new stuff that I hadn't gotten around to yet.
First up was a Sans Digital MS2T+B MobileSTOR enclosure I had picked up a while back. The current array that backuppc goes to on orac is filling up. It's just a 4TB filesystem on a RAID6 array using 1.5TB drives. There's one other filesystem on this array and it could use some more room as well. I have debated whether the array needs to be RAID6 for backuppc, or whether RAID5 would be sufficient. But its the other filesystem that is driving RAID6, though I wonder if RAID 0+1 might be better for that (or just RAID1, since it is currently small enough). But, I'm using a pair of 5 bay PM enclosures....and older RAID 1 in one, and the RAID6 array in the other.
So, I've decided that I'll likely look at getting 5 2TB drives and doing RAID5 for a backuppc only array....keep the current 5 1.5TB drives in RAID6 for the other filesystem and whatever else I might want to move over to it. But, the stuff on the current 1TB RAID 1 array should just stay put. Well, there are two (of 4) regular eSATA ports on the back of orac that aren't being used yet. So, that's where the 1TB RAID 1 array will go.
It is still a ways off before I take the plunge and actually build the new array (though the backuppc filesystem has filled up twice since I thought up this scheme)... though prices are dropping and I do have a tax refund coming that many other things I want to use it on (including a travel/backup cpap setup, which is about the entire refund...oops).
But, I decided I would set it up, because I saw 2TB Hitachi drives the other day for cheap (they did recently get acquired by Western Digital, so sellers might be clearing out the Hitachi branded drives?) So, it might fall into range of what would be an acceptable single month discretionary purchase. Though technically, I've already done this month's when I bought the Nanosat speaker package that was a NewEgg shell shocker....though I used BillMeLater. I've been wanting to get this set to upgrade the odd ball collection of old speakers that is my current setup. But, the shellshocker price put it just into the range.... But, I was considering Seagate drives...which then dropped to remain cheaper than Hitachi....Hmm....
Anyways...I went to put the new enclosure into service, but I couldn't find the pair of eSATA cables I thought I had set aside for this. I did buy a pair of eSATA cables last fall from Amazon.com. But, evidently I used them for something else. I'd guess when I moved orac, I needed the length...not sure where the old cables went to then, but the probably aren't long enough for this. So, did some quick eBay purchases...and maybe I'll get to this eventually....
Its time that the full backups have been taking place on orac's backuppc. And, my /var/lib/backuppc doesn't quite have enough free space for a full of some of the jobs (I guess it doesn't matter if many, if not all, of the files would already be in the pool, it still needs more free space than I have in the filesystem to do the job).
So, I was looking at what I could about moving some stuff around so that I can grow /var/lib/backuppc by more some more....to perhaps 3.5TB. And, I thought about why it was that orac was located where it was. Basically it was just an open spot near the monitor.... And, it really needed to move somewhere, where I could better park the external drive cases. Currently the main one was a Rosewill RSV-S5...there's a Sans Digital TR2UT-B, but I had recently picked up a Sans Digital TR5M-B that I needs a place where I can park it near orac.... Along with the two old 500GB My DVR Expanders.
So, the plan was to move it next to lhaven, though I ended up moving lhaven into the spot opened up by orac. orac has side vents so it couldn't go exactly where lhaven was, but decided to move lhaven out of the way so that I could get in and cable up orac.
Before turning orac back on...I did the storage changes. I setup the Sans Digital TR5M-B, its smaller than the Rosewill RSV-S5....moved the 2 1TB drives from the Sans Digital TR2UT-B over to it, and hooked up the 2 old 500G MyDVR Expanders to the regular eSATA ports on the back of orac (the system has 6 SATA ports internally, and I had gotten 4 port SATA back to run 4 of the ports out for future upgrades....that was before I got the 2 port PM eSATA adapter for the PCIe x1 slot.)
Also discovered that the bays number the other way around in the TR5M.
Now with orac up again, I was kind of expecting the drives to be in different places...but they were all in the same place. And, 500G MyDVR Expanders in sequence after the two internal drives.
Quick look around, and then immediately went about turning the MyDVR Expanders into a RAID 1 set and a single ext4 filesystem. And, then I started moving files over to it. Of course, that slows down the build of the RAID....
But, now to wait until I decide how to move the other pieces around....
Wonder how long before I amass the 5 2TB drives to build the new RAID5 to move the current /var/lib/backuppc over to.... (keeping the current 5 1.5TB RAID6 for more important data).
Ever since I learned about 5-bay port multiplier SATA enclosures, I've been wanting to get one to play with. And, when I acquired 'orac'...I sought out a port multiplier eSATA adapter to include in the build of the system. And, then waited for support of the adapter in ubuntu. Stopped on 8.04LTS and waited for inclusion in a kernel.
I took a step closer when I finally got the 2-bay enclosure to mirror a pair of drives I had impulsively purchased. Since it worked, I then then bought a 5-bay enclosure (it came up as a NewEgg ShellShocker).
But, after some thought, I decided that what I wanted to do was build a 5 disk RAID5 using 1.5TB drives....so it would get me at least 5TB of real storage. And, set that I would wait until 1.5TB drives were under $100.
In the meantime, I collected a bunch of old SATA drives to see what I trouble I could get into....it didn't last long, as one was definitely dead and two quickly failed. Of course, two of the had been kicked out of RAID1 setups previously, and the other was a failing drive that I replaced/upgraded. So, it was back to waiting.
Eventually, I got 3 Samsung 1.5TB drives through NewEgg ShellShocker. And, I set up a 3 disk RAID5. Sadly, it didn't last very long....one of the drives failed while I was building the array. I exchanged it for a new drive with NewEgg, and waited for it before trying again. While I was waiting, I formatted it a bunch of times and learning about smartctl and various things of drive repair. One of the remaining two original drives, was showing signs of slow death and one unrecoverable sector. When, the new drive arrived...I did a long self test and a quick, but full, format....taking a couple days before I created the 3 disk RAID 5.
Once, that was done, I created a new big filesystem that is /var/lib/backuppc to have backuppc on 'orac'. Initially a 1TB filesystem.
Then one day, the bad happened. A drive on 'gumby' failed. It was time to stop putting off figuring out how to setup backuppc and start putting stuff into it. It turned out to be a lot easier than it thought to get started, though I did have to rebuild smbclient to have a longer timeout to get it successfully complete a full backup of a given partition. I then learned some other stuff and fine tuned the configs.... One big change was I broke out the windows hosts into individual partitions (ie, gumby_c, gumby_e, gumby_f), and came up with a 'semaphore' solution so that it wouldn't backup more than one partition at a time from a windows host. Eventually, I had all my systems in it (it fully backs up everything except itself, where it backs up only key parts....) I even got it backing up my work Mac Book Pro.
I later grew the filesystem out to 2TB.
Pages: 1· 2
The other day, as I was rebuilding my 5 disk RAID array (which I mean to blog about someday)...for the umpteenth time. A storm came in one morning....this made me concerned about losing power and breaking my RAID in some bad state (it had happened once before). Now I did have the computer and its two disk arrays (and as it turned out monitor) on a Geeksquad 1285VA UPS. And, more serious now that I have quite a lot of data backed up onto this array using backuppc. Had gone through some scary periods where I thought I'd have to start over from scratch.
But, decided that may it was time to put it on a PowerSource 400, like my other two Linux servers. So, I went to my usual source....not available from there, but it listed 3 other sellers; two selling for substantially more than it used to sell for and one selling refurbished units. I went looking around online for other sources, including one other that I had considered as an alternative option (because they shipped using FedEx ground). Found that pretty much everybody that had listings for it, showed it as discontinued and not in stock. I check eBay, there were a few listings...but they were mostly of the store demo variety....seemed too risky for me, given past experiences.
So, I put it in my cart for the next time I was going to place a home bound Amazon Prime order (a PowerSource 400 is a heavy item, so even if I don't know when it'll arrive...I wouldn't want to be lugging it back from the UPS Store).
As it happened, the seller shipped it by UPS Basic, and the mail man showed up with it on the day that I stayed home sick....ran the bell twice and then left it on my door step (took me a while to unmask, get out of bed and find some pants). I didn't immediately set it up....but found somewhere to plug it in to at least charge. I was fully discharged, so it would take a while.
Later that evening a big storm came into the area, alert said golf ball sized hail....it sounded pretty intense out there. Just had to happen as I was trying to do the UPS swap. Discovered some oddities about how I have things run for power around my desk that I had forgotten about. But, I think it is all sorted out now. Though I may opt to move the monitor to another UPS at some later point.
Currently, monitor, server (orac) and the two raid boxes (2 drive - RAID1, 5 drive - RAID6) are on this 'new' PowerSource 400 and its reading 250-260W.
Annoyingly the literature doesn't give an estimate for loads >100W on the PowerSource 400....so I'm not really sure how much time I'll get at this...but hope its something. For comparison. 'lhaven' on its PowerSource 400 is reading 100-110W, and 'box' and the network gear (5 port gigabit switch and g access point) on its PowerSource 400 is reading 150-160W. I may pull the access point off the UPS at some later point in the future, since it is deprecated as I mainly use my Airport Extreme now...which is in another location and on a PowerSource 400.
That other PowerSource 400 is reading ~130W...it has 2 TiVo HDs, both with DVR Expanders, router, 16 port 10/100 switch, and the Airport Extreme. I was originally planning to add a RAID USB storage device to the Airport Extreme...though not sure now...that I have my work Mac Book Pro set up as a client to backuppc. (it also rarely comes how now.)
I don't think I have the courage to test how long any of my PowerSource 400s will actually last....but so far I've been happy with the ones that have endured the few extended outages I experienced last year.
I wonder what will be the successor to the PowerSource 400....and whether there'll be more PowerSource 400s, or what, in the future for me.
Pages: 1· 2
box is such an important server to my home network, while the last time it failed wasn't due to drive failure...there had been times in its past where it was.
And, since I was playing around with RAID a lot on orac, I decided that it was probably time that I convert box to do RAID.
It doesn't need a lot of storage....it has always been a 40GB harddrive of some sort. That had come either as stock in the off-lease system I built it around or from a ReplayTV. The latest incarnation of 'box', a Thinkcentre S51, used SATA drives, so spare drives wasn't something I had a pile of for it.
Now being a Thinkcentre S51, there wasn't much in the way of internal options of doing RAID, but there was a second SATA port on the motherboard. And, it did have USB ports...so at first I was looking at creating RAID using an external harddrive. Either USB or SATA. Leaning towards SATA (had purchased a SATA bracket).
But, I was looking for a pair of small SATA drives....at first one the same size as the current, but then a pair of small/same would do....and more procrastination.
But, then the K-State Student Union computer store had a clearance table out...and on it where various sizes of 2.5" SATA drives. And, I recalled seeing an adapter that would allow me to mount a pair of 2.5" SATA drives in a 3.5" drive bay. There was even a version that had on board RAID (and could do USB)....similar to one of the RAID enclosures connected to orac. Though I wanted to go with mdadm managing my raid in software, since it was a little bit better known to me in its setup, management and recovery. The latter is kind of important, since at work we've had fun in trying to recover systems using hardware RAID after a card/board swap or having it tell us its status by some mean other than wondering if a light shouldn't be on or something while passing by it in the data center.
But, the table had come out before I was going on my winter holidays...so I wasn't in a position to figure things out and buy it. Later when I came back from holidays, I saw that some of the 2.5" drives were still available...and had been marked down even further. The big ones were gone, and a co-worker mentioned that he had gotten a pair of 250G for just such a use. There was a single 80G and 3 120G left. So, either I could get a single enclosure and go with the old external mount of drive plan....or get a pair of 120G drives and explore the 2x2.5 to 3.5 bay adapter route. I ended up buying all 3 of the 120G drives, figuring that I'd have a cold spare for down the road.
I then checked eBay for the adapter....and bought the one that wasn't RAID or anything. I chose a seller that shipped USPS, and it just made it in on New Year's Eve.
During the thinking and research phase, I found the linked to details on how to do what I wanted.... And, I pretty much used it (with minor adjustments to suit my needs, like the differences of my ubuntu release, or that I was going to swtich to having /home on /dev/md1. Or that I wasn't going to reuse the old drive in the conversion, etc.
Pages: 1· 2
Was discussion storage speeds with some friends over lunch....so I wondered if I could get some data comparing the performance.
vg0 is are two internal SATA drives, each on their own SATA channel, being mirrored. vg1 is the SANS Digital pair of drives, sharing a SATA channel via Port Multiplication, being mirrored.
Guess the graphs show that the internal drives are faster for small files....though for larger the difference goes away, so that things are comparable.
As it is....vg1 is mainly my MMCVideo filesystem - MMC (Momitsu Media Center). So, the files tend to be large and sequentially read/written.
Meanwhile, I've already removed the old MMCVideo logical volume from vg0 and doubled the size of /home. Even though now that I think of it, there's stuff under /home that might better go in its own logical volume(s)....now that I have room to create them.
Or later go on that HP MediaSmart Server that I've been dreaming of adding to my network....
Latest Poopli Updaters -- http://lkc.me/poop
|<< <||> >>|
10.04lts lhaven staples dvd tivo raid dsl «tivo hd» cfengine3 prescription eyeglasses virtualbox «chicago tardis» tardis «doctor who» «tivo premiere» «air purifier» woot migration «watch instantly» usb b2evolution raid1 «hd movie» «powersource 400» box freebsd amazon.com «windows xp» tv upgrade ups backuppc orac cox cpap ebay ubuntu boinc «instant streaming» «windows 7» mdadm «amazon prime» netflix «sans digital» twitter replaytv linux zen appletv