Archives for: July 2013

07/29/13

  09:06:00 pm, by The Dreamer   , 729 words  
Categories: Operating Systems, FreeBSD, CFEngine

The risk of high uptimes....

There are Unix servers at work that have uptimes in the >1000 days, there are even servers with updates in the >2000 days, in fact there are servers that have now exceeded 2500 days (I'm looking at one with 2562+ days.)

On one hand there are SAs that see this as a badge of honor or something to have had a system stay up this long. OTOH, its a system of great dread.

A while back this system was having problems....its Solaris and somebody had filled up /tmp....fortunately, I was able to clean things up and recover before another SA resorted to hard rebooting it.

The problem with these long running servers, especially in a ever changing, multi-admin shop, is that you can't be sure that the system will come back up correctly after a reboot.

We've lost a few systems at work due to a reboot. Some significant ones as simple as replacing a root disks under vxvm and forgetting to update the sun partition table, or a zpool upgrade and forgetting to reinstall the boot. To more significant ones, where a former SA had temporarily changed the purpose of an existing system all by command line and running out of /tmp...so that after its been up for 3+ years and he's been gone over a year....patching and rebooting makes it disappear.... the hardware that the system was supposed to be on needed repair, but he had never gotten around to it.

It'll be interesting to see what happens should the system ever get rebooted.

:?: So, what brought this post one?

Full story »

07/28/13

  01:14:00 pm, by The Dreamer   , 3144 words  
Categories: Software, Operating Systems, Ubuntu, FreeBSD, CFEngine

Last two weekends - nagios and more cfengine 2 & 3

So, what started as take a week to set up a new nagios server at work ended up taking almost a month...because there were many days where I'd only have an hour or less to put some time into the side task. The other stumbling block was I had decided that the new nagios server configuration files would get managed under subversion, instead of RCS as it had been done in the previous two incarnations. New SA's don't seem to understand RCS and that the file is read-only for a reason...and its not to make them use :w! ... which lately has resulted in a the sudden reappearance of monitors of systems that had been shutdown long ago.

Though now that I think of it, there used to be the documented procedure for editing zone files (back when it was done directly on the master nameserver and version controlled by RCS.) Which as I recall was to perform an rcsdiff, and then use the appropriate workflow to edit the zone file.

% rcsdiff zonefile

if differences

      % rcs -l zonefile
      % ci -l zonefile
        make rude comment that somebody made edits
      % vi zonefile
      % ci -u zonefile

else

      % co -l zonefile
      % vi zonefile
      % ci -u zonefile

fi

But, when I took over managing DNS servers, I switched to having cfengine manage them and the zone files now live under masterfiles, so version control is now done using subversion. Had started butchering the DNS section in the wiki, probably should see about writing something up on all the not so simple things I've done to DNS since taking it over...like split, stealth, sed processing of master zone for different views, DNSSEC, the incomplete work to allow outside secondary to take over as master should we ever get a DR site, and other gotchas, like consistent naming of slave zone files now that they are binary.

Additionally work on the nagios at work was hampered by the fact that for Solaris and legacy provisioning is CF2, and the new chef based provisioning is still a work in progress...where I haven't had time to get into any of it yet. So, I had to recreate my CF3 promises for nagios in CF2.

But Friday before last weekend it finally reached the point where it was ready to go live. Though I've been rolling in other wishlist items and smashing bugs in its configuration, and still need to decide what the actual procedure will be for delegating sections of nagios to other groups.

One of the things I had done with new nagios at work, was set up PNP4Nagios...as I had done at home. And, while looking to see if I needed to apply performance tweaks to the work nagios, all the pointers were to have mrtg or cacti collect and plot data from nagiostats. Well, a new work cacti is probably not going to happen anytime soon, and the old cacti(s) are struggling to monitor what they have now (I spent some time a while back trying to tune one them...but its probably partly being hampered by the fact that its mysql can use double the memory that is allocated to the VM. though reducing it from running 2 spine's of 200 threads each...on the 2 CPU VM to a single spine with fewer threads has helped. Something like the boost plugin would probably help in this case, but the version of cacti is pre-PIA. But, it could be a long time before it get's replaced (not sure if upgrade is possible....) Our old cacti is running on a Dell poweredge server that has been out of service over 6 years... with the cacti instance over 8 years old (Jul 8, 2005)....and the OS is RHEL3.

Anyways, it occurs to me that there should be a way to get PNP4Nagios to generate the graphs, and I search around and find check_nagiostats. Though no template for it. Oh, there's a template nagiostats.php, if I create a link for check_nagiostats.php it should get me 'better' graphs. Which is what I have CF2 do at work.

Full story »

Pages: 1· 2· 3

07/14/13

  05:58:00 pm, by The Dreamer   , 2452 words  
Categories: Software, Ubuntu, FreeBSD, CFEngine

Moving irssi

So, recently there was a 'long' 4th of July weekend....on account that I opted to take Friday (the 5th) off as well.

I kind of thought I would tackle a bunch of different projects this weekend, though I've pretty much shelved the idea of re-IP'ng my home network. Perhaps something to do when I get my configuration management better fleshed out.

What I decided was that it looks like its just one last thing on one of the two Ubuntu servers that I'm retiring. So, I figured I'd quickly move that and then go onto the next thing. In the end, I didn't get it completed until Monday night.

For background, some years back...after my return to IRC, I had initially gone with Chatzilla (being that Firefox was my standard browser), which later moved to xulrunner and Chatzilla so it was independent of my browser. Though it was kind of annoying having it running at work and at home, and somewhat confusing for co-workers that ran text based IRC clients in screen somewhere and ssh'd in, etc. Most people that did this, were doing irssi.

So, I initially built it from source and was running on my old RedHat 7.3 server, and that was usable. Later when I setup an Ubuntu box to replace that server (the hardware had previously been SuSE....acting as an internal router for ivs status tracking....) It evolved, in that I would start screen detached from rc.local....which was important since the system would see patches on a regular basis, requiring reboots....which is kind of a reason for switching to FreeBSD.

Over time, I would make little tweaks here and there, to this irssi setup. Like twirssi, doing ssl, and later bitlbee to integrate Facebook chat (came across some stuff that I should add now...)

And, incorporating other tweaks I come across online when there's some problem that becomes sufficient bothersome that I want to address. The one problem I haven't haven't been able to solve is keeping server/system messages confined to the one window. Namely keeping system CRAP going to the system window, and allow channel CRAP to show up in the channel windows....but instead I'll get system CRAP in whatever channel window is active. Which is annoying because its usually the work channel. Where it be just signal and no noise.

Anyways...

I had started to move things more than a month ago, in that I built irssi and bitlbee (including the cfengine3 promise for it...not really much config wise for cfengine to manage for irssi...though I envisioned promising that its running all the time, though irssi has generally been stable everywhere else that I've run it.

But, the I got distracted by other cfengine3 work. Even though things started to get pressing when twirssi stopped working, due to API 1.0 going away...so I had to update Net::Twitter and twirssi. Updating twirssi wasn't that hard to do, but Net::Twitter was a problem, so I opted to remove it and its dependencies and then installing it and its dependencies using CPAN.

I also made note to install net/p5-Net-Twitter from ports on dbox.

twirssi seems to be having other issues, which I had intended to investigate...perhaps after I move... But, that was like a month ago....

Full story »

Pages: 1· 2

07/06/13

  09:32:00 am, by The Dreamer   , 659 words  
Categories: Software, Ubuntu, FreeBSD

Ubuntu squid with SSL

Link: http://lawrencechen.net/ddclient-aamp-squid

This is an update to the "ddclient & squid" here

Ran into a new problem recently....though the need for SSL in squid on ubuntu is deprecated, by the fact that I'm slowly replacing this server with a FreeBSD server.

As a result, I don't pay attention to this ubuntu server as much as I used to, so I've configured unattended-upgrade. It was installed, but it didn't seem to do anything in that on other servers I'd log in to find that there are lots (40+) of patches available and more than half that are security. Since I came across how to configure it to do more than just security patches, including send me email and on some systems automatically reboot when necessary. (should've thought to see how unattended-upgrade is configured and doing such things in the Ubuntu AMI I have in AWS)

Since I got unattended-upgrade configured on this old server (32-bit Ubuntu Server, which I've heard they have a 12.04LTS download for??? They had said they dropped 32-bit server support, so there was version with 10.04LTS. So I couldn't upgrade and now I'm way past EOL, which is causing problems...probably need to hunt down the landscape and ubuntuone services and nuke them, instead of letting them degrade my server for being EOL.) I've also had to update packages on here from outside sources to keep things running, so guess I should work harder on abandoning this server.... Where it'll likely get reborn as [yet ]a[nother] FreeBSD server....along with the server that I think I have all the parts collected for it, but just need to sit down and put it together. It started as a mostly function pulled 1U server, in need of ... well either new fans or a new case.... I opted for the new case route. It also needed drives and memory. But, as a result of the new case route...aside from case/powersupply...it meant I would need to get heatsinks...since the passive ones based on the 1U case channeling air flow....would be hard to recreate in the tower case I went with. Its a huge tower case, given that its an E-ATX motherboard...yet it isn't a full tower (like the formerly windows machine called TARDIS...someday I'll work its regeneration....need money to buy all the bits and pieces that'll make that up, which I haven't fully worked out what those will be....or where it'll go since my dual 23" widescreen FreeBSD desktop has consumed all of the desk that it would've shared....and not really keen on the idea of a KVM for this situation. :hmm: )

Anyways...every day I get an email from unattended-upgrade for this system.... with:

Unattended upgrade returned: True

Packages that are upgraded:
 squid-common 
Packages with upgradable origin but kept back:
 squid squid-cgi 

Package installation log:


Unattended-upgrades log:
Initial blacklisted packages: 
Starting unattended upgrades script
Allowed origins are: ["['Ubuntu', 'heron-security']", "['Ubuntu', 'heron-updates']"]
package 'squid' upgradable but fails to be marked for upgrade (E:Unable to correct problems, you have held broken packages.)
Packages that are upgraded: squid-common
Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg_2013-07-06_08:05:42.056193.log'
All upgrades installed

This is because of that quirk where even though I rebuilt my version with SSL, and kept it the same version...it wants to install its version to replace mine (of the same version). Which is why I did the hold thing.

I could do the alternative of add a string to make my version advance from current....though I suppose I won't unhold...so that unattended-upgrade won't upgrade should such a thing appear (unlikely since both the OS and squid are ancient...and there'll be no more updates.) But, the intent is to hopefully silence unattended-upgrade in this matter.

Though kind of surprised its still doing something....hmmm, guess there was a new security patch to squid 2.7 back on January 29, 2013....that I've been missing (suppose its already downloaded the update in its 'cache'....or the backend is still there, its just not getting updates beyond what's there....whatever, I think I'm down to one more service to move off....)

07/04/13

  05:59:00 pm, by The Dreamer   , 476 words  
Categories: Software, CFEngine

Upgrading from CFEngine 2 to CFEngine 3

I just learned of a key missing detail that would probably have helped lots of other CFEngine 2 sites make the transition to becoming CFEngine 3 sites.

All the sites, include CFEngine's have docs about Upgrading from CFEngine 2 to 3....

Where, the touch, or go in-depth, on conversion of policies from 2 to 3, extol how 3 is better than 2, and then offer vague options on how to upgrade (either in-place or replace)....

The most detailed explanation was a slide deck...which wasn't detailed enough.... that says "CF2 and CF3 designed to be interoperable", "Replace CF2 policies at your pace". How?

In-Place Upgrade

"Replace cfexecd with CFEngine 3's cf-execd" - Access controls remains untouched, runs cf-agent.

"Sample input files contain integration promises" - Launched automatically, Changes crontab

And, then get's in the steps:

  • Install CFEngine 3
  • Copy new inputs files to CF2 master repository
  • Remove any rules to reinstall CF2 or add cfexecd or cfagent to crontabs
  • Remove cfexecd from start up
  • Edit update.cf
  • set email options for executor in promises.cf
  • cf-agent --bootstrap

If all went well, you are now running CFEngine 3.

Bootstrap policy server using:

cf-agent --bootstrap --policy-server

  • Remove all rules and policies that are capable of activating CFEngine 2 components
  • Convert cfservd.conf into a server bundle
  • Place a reference to this in promises.cf
  • Add converted CFEngine 2 policies or create new CFEngine 3 policies
  • Done???? :??:

    Somethings missing....where's this interoperability taking place? Does CF3 know how to run CF2 policies? no... where's this replace CF2 with CF3 at my pace? Reads like its a full in-pace replacement of CF2 to CF3....

    So I finally made a reference about this on a list...

    Answer?!

    It's why the CF3 binaries have dashes in the name. So you can drop them into the CF2 working directory.... The trick is editing the exec_command in the executor configuration, that's the command for running the agent; modify it to run both agents (v2 and v3).

    Wow...that's kind of an important detail that's been missing!

    Full story »

      02:58:00 pm, by The Dreamer   , 254 words  
    Categories: Software, FreeBSD, CFEngine

    hindsight on cfengine 3

    In retrospect, maybe what I should've done is switched the origin of my sysutil/cfengine to sysutil/cfengine34 when 3.5.0 came out. Since, I see that cfengine-3.4.5 has recently come out, bug fixes to cfengine-3.4.4 were more of what I was after than new features. Though I am intrigued by what 3.5.0 appears to bring, and am considering making use of it...of course, by the time I get to it 3.5.1 or newer might be out.

    OTOH, do I really want to build cfengine-3.4.5 in semi-usable package management system we use at work for building and maintaining packages for Solaris 9 and Solaris 10 SPARC, and Solaris 10 x64. The system builds everything 32-bit, though I'm pretty sure we don't have 32-bit hardware anywhere in the datacenter anymore. Though we still have a few Solaris 10 systems around.

    Hmmm....

    % wget 'https://www.cfengine.com/source-code/download?file=cfengine-3.4.5.tar.gz'
    --2013-07-04 08:39:34--  https://www.cfengine.com/source-code/download?file=cfengine-3.4.5.tar.gz
    Resolving www.cfengine.com (www.cfengine.com)... 62.109.39.150
    Connecting to www.cfengine.com (www.cfengine.com)|62.109.39.150|:443... connected.
    OpenSSL: error:14077458:SSL routines:SSL23_GET_SERVER_HELLO:reason(1112)
    Unable to establish SSL connection.
    

    :hmm:

    Seems to be a problem with a client using openssl 0.9.8 talking to a webserver using 1.0.0?

    Guess there's a patch submitted against 0.9.8y.... http://www.mail-archive.com/openssl-dev@openssl.org/msg32486.html

    But, this will be a big mess at work....nothing is using 0.9.8y yet (though I've been meaning to build it so I'll be ready when there's a bind-9.9.3-P2...had started building 9.9.3 when there was a security advisory of problem introduced in that version...so I'm waiting for the next 'real' security patch to do the upgrade...though maybe I shouldn't, since the intent is for this to be the first 64-bit build....)

    Not sure what I'm going to do about cfengine3 at work though....

    Now instead of subjecting some poor random forum to a long rambling thought, I will try to consolidate those things into this blog where they can be more easily ignored profess to be collected thoughts from my mind.

    Latest Poopli Updaters -- http://lkc.me/poop

    bloglovin

    There are 20 years 1 month 24 days 12 hours 29 minutes and 59 seconds until the end of time.
    And, it has been 4 years 11 months 3 days 1 hour 32 minutes and 57 seconds since The Doctor saved us all from the end of the World!

    Search

    July 2013
    Mon Tue Wed Thu Fri Sat Sun
     << < Current> >>
    1 2 3 4 5 6 7
    8 9 10 11 12 13 14
    15 16 17 18 19 20 21
    22 23 24 25 26 27 28
    29 30 31        
    Google

    Linkblog

      XML Feeds

    Who's Online?

    • Guest Users: 1
    This seal is issued to lawrencechen.net by StopTheHacker Inc.
    multiple blogs

    hosted by
    Green Web Hosting! This site hosted by DreamHost.

    monitored by
    Monitored by eXternalTest
    SiteUptime Web Site Monitoring Service
    website uptime