« VLANs & FreeBSD 9.0/JailsGetting nxserver working on FreeBSD »

Orac is looking strangely bare, with Zen taking over.

11/05/12

  10:30:00 am, by The Dreamer   , 2100 words  
Categories: Hardware, Software, Computer, Storage, Ubuntu, FreeBSD, VirtualBox

Orac is looking strangely bare, with Zen taking over.

Like Orac, I had gotten a pair of 5-bay PortMultiplier enclosures for it to hang off of the 2 port eSATA adapter, initially a SIL3132 and later the RR622...while Orac stayed with a SIL3132 adapter. The RR622 worked better under Windows than the SIL3132...but the SIL3132 works better under Ubuntu (and FreeBSD). I found the RAIDZ array to be really slow on the RR622, and it was taking several minutes to boot. In the past disabling RAID and switching to AHCI would fix the long boot times, but reflashing the RR622 with non-RAID firmware didn't help. Eventually, I tracked down the old SIL3132 adapter I had in this computer and boot times improved considerably as did throughput with my RAIDZ array (like 10x).

Aside from running out of space in the 5.4TB BackupPC array on Orac, I was also having a problem with ext4 filesystem corruption, periodically resulting in reboots and manual intervention to fsck the filesystem back...resulting in portions of the cpool getting lost due to filename corruption and other errors. It was making me contemplate redoing it and trying ZFS under Ubuntu again. But, now that I have a FreeBSD machine, I could do ZFS there. Since I got BackupPC running on my FreeBSD system at work, I tried to do the same at home. I had gotten it working at work under nginx, and a fastcgi server running as 'backuppc'. It works under Ubuntu using perl-suid & Apache, but that isn't available under FreeBSD. Eventually, that failed...so I settled on Apache and SUEXEC to make it go. I created a filesystem for it on the RAIDZ to test things out, and pretty soon I had added all the host definitions to it....knowing that there was no way to cram all the data into the free space of the RAIDZ set. It topped out at ~2.5TB of full backup data crammed into the ~1.75TB of free space in the RAIDZ pool. And, still a lot of systems not backed up. Though there's a couple of host definitions that may never get backed up, since those machines are gone. Had wanted to keep them, for eventual recovery...though when I had tried a restore of one of them, it wasn't 100% successful due to pool corruption over time. So, it the other old backups were likely impacted too.

So, having filled the filesystem on Zen, and deciding that I would continue this change. I exported the lone 1.5TB disk....so that I would have 6 open bays on Zen....and then move the 6 drives that make the RAID10 on Orac over to Zen. They're all Sans Digital 5-bay arrays (though one on Orac is a Rosewill clone....), though Zen has one TR5M-B and one TR5M-BP on it....had been using the TR5M-BP alone at first...but now with adding 6 more drives...the TR5M-B is in service. Not sure what's different in the enclosures to go from 3Gbps to 6Gbps, so I don't know if there would be any benefit in replacing the SIL3132 (3Gps) with something that does 6Gps (thinking an ASM1061 based adapter, since from browsing the ahci driver source, it shows at not requiring any quirks... the SIL3132 being handled by the SIIS driver, might mean there's no reason to leave it? Doubt that I would get any more speed out of things...think I'm not saturating the 3Gbps...so extra headroom probably won't help. Though the Hitachi drives are reporting as 6Gps. Though having AHCI for internal and SIIS for external, means the internal drives show up first and stay with original mapping...(ada0,ada1,ada2) and SIIS drives fill in after...though the ordering doesn't match the target orders....not sure why. Only caused an issue because I have a swap partition, and was mounting as ada0p1...switching to mounting as gpt/swap0 fixed that. ada0 is an SSD drive. Also using a portion of it as L2ARC for the internal mirror pool. Thinking another SSD or two is in my future now, so I can add some L2ARC to the RAIDZ and RAIDZ2 pools.

Anyways...when it was the RR622, they were all ahci...and it had the external drives mapping first.... and the ASM1061 is also under the AHCI driver. The "Intel Series 5/3400 Series AHCI SATA Controller" on the motherboard doesn't have quirks set, while the RR622 has AHCI_Q_NOBSYSRES, the ASM1061 also has none set, while the Marvel 88SE9128 has AHCI_Q_NOBSYSRES|AHCI_Q_ALTSIG set. Though not sure if these are good or bad quirks, just that the RR622 was poor for me compared to the SIL3132. I think I have a Marvel based adapter somewhere in my collection, in fact I think it is what I was using before the RR622....and in Windows the RR622 was much better than the Marvel one. I suppose I'll just stay with the SIL3132.

I suppose the question is what I should use internally for drive some more SSDs....I have a SIL3124 card, or should I find an ASM1061 based card instead.

Anyways...I set up the 6 drives on Zen as a RAIDZ2 (I considered it just as RAIDZ to maximize space, but decided that I still wanted the extra redundancy....throughput is still quite good. I did a zfs send/receive to move the exisiting BackupPC pool from the RAIDZ to the RAIDZ2, and it was doing about 100MBps.... later I did a scrub of both pools...I was seeing almost 100MBps on the RAIDZ with 1.5TB 7200RPM drives, and around 110MBps with the RAIDZ2 with 2TB 5940RPM drives.

When I did a scrub of the pools I was seeing peeks of 100MBps on the RAIDZ with the 1.5TB 7200RPM drives, and peaks of just over 110MBps with the RAIDZ2 with the 2TB 5940RPM drives. Hopefully the 7.2TB will be enough until 3TB drives finally drop under $100 (or perhaps 4TB drives...) Though it might be a problem in that the Hitachi 2TB drives are 512-byte sector drives. And, not sure how what the shuffle would look like. I don't have the slots to add more storage to Zen....well, I suppose if I take out the USB 3.0 card.... :hmm:

Wonder how long it'll take to fill up the 7.2TB array? Right now there's more data on Zen then there was before, since its a bunch of different systems combined...but a lot of it dupes other places so it shouldn't take up much more space. Though I got pyTiVo running on it, so there's yet more TiVo content waiting for when I get around to catching up on my shows.

But, I think I want to refill the open bays on Orac first....and get BackupPC going again over there. Have it backup Zen, instead of Zen backing itself up. (though I was having Orac backing itself up before, though it was challenging when I went from 8.04LTS to 10.04LTS and some of my arrays were lost...which is kind of why they don't start telling you that 12.04LTS is available as an upgrade until 12.04.1 is released now. Since it turned out to be an mdadm bug in 10.04 that was likely fixed a week or so after I had made the move. :##

But, now I'm holding off on moving from 10.04LTS to 12.04LTS, since I've decided that I want to move the core services off of my Ubuntu servers to FreeBSD (which calls for building two more systems ?) Orac is still an Ubuntu Server install, but Box being only 32-bit hardware upgraded into a Generic desktop with 10.04LTS. But, its more of a server than Orac is. Though I've thought about putting an Nvidia card into it...though that's more for BOINC than to run a Desktop on it. Also, need to make the switch so I can try switching to DVI-D KVM and make use of Nvidia card in Orac again. Someday I should resurrect Gumby, and don't know if LHAVEN needs to join the KVM, though I have the old GT520 from Zen that I could put into it.

Couldn't get the GT520 to work with Ubuntu 12.04LTS, was apparently a known issue in its initial release with Nvidia cards with more than 1GB of memory or something. Wonder if its fixed now....though I also had trouble with the GT520 under FreeBSD, so after several cards...I ended up with an FX1700 that works. Which might also work for Ubuntu, except the ZFS is great. The only trade off is VirtualBox lacks USB 2.0 support on FreeBSD, and its kind of an important thing with running Windows anywhere. Right now those USB devices connect to a Windows XP VirtualBox on an Ubuntu host...but I'm working on recovering my Windows 7 environment on Zen, and eventually the lack of USB 2.0 might become a problem. But, I have a few ideas on how to overcome that....

Pages: · 2

No feedback yet

Now instead of subjecting some poor random forum to a long rambling thought, I will try to consolidate those things into this blog where they can be more easily ignored profess to be collected thoughts from my mind.

Latest Poopli Updaters -- http://lkc.me/poop

bloglovin

There are 20 years 3 months 23 days 12 hours 45 minutes and 5 seconds until the end of time.
And, it has been 4 years 9 months 4 days 1 hour 17 minutes and 51 seconds since The Doctor saved us all from the end of the World!

Search

September 2017
Mon Tue Wed Thu Fri Sat Sun
 << <   > >>
        1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30  
Google

Linkblog

  XML Feeds

Who's Online?

  • Guest Users: 4
This seal is issued to lawrencechen.net by StopTheHacker Inc.
Open Source CMS

hosted by
Green Web Hosting! This site hosted by DreamHost.

monitored by
Monitored by eXternalTest
SiteUptime Web Site Monitoring Service
website uptime