Category: "CFEngine"

Pages: 2

06/08/13

  08:42:00 pm, by The Dreamer   , 1176 words  
Categories: Software, Computer, Storage, FreeBSD, CFEngine

Another weekend seems to be slipping away on me....

And, its the same time suck....cacti.

Last weekend got away from me, because I to make another attempt to improve cacti performance. I had tried adding 3 more devices to it, and that sent it over the limit.

I tried the boost plugin....but it didn't help, and only made things more complicated and failure prone. Evidently, updating rrd files is not a constraint on my cacti server. Probably because of running on an SSD.

I made another stab at getting the percona monitoring scripts to actually work under script server, but that failed. I suspect the scripts aren't reentrant, because of their use of global variables and relying on 'exit' to cleanup things it allocates or opens.

I had blown some previous weekend when I had tried to build the most recent version of hiphop to maybe compile the scripts, but after all the work in figuring out how to compile the latest 2.0x version...it would SEGV, just as the older lang/hiphop-php did after resolving the problem of building with the current boost (a template had changed to need a static method, meaning old code won't link with newer boost libraries without a definition of this.) And, this is beyond what I have in my wheelhouse to try to fix.

During the week, I had come across some more articles on tuning FreeBSD, namely a discussion of kern.hz for desktop vs servers. Where it being 1000 by default is good for desktops, but the historical setting of 100 being what to use for servers. Though IIRC, ubuntu uses 250 HZ for desktops and 100 HZ for servers, it also doesn't do preemption in its server kernel along with other changes (wonder if some of those would apply to FreeBSD?) Though modern kernels have been moving to be tickless. Which I thought was in for FreeBSD 9, though the more correct term is dynamic tick mode...and which is more about not doing unnecessary work when things are idle. Which isn't the case with 'cbox'. So, perhaps, fiddling with kern.hz and other sysctls might still be relevant. Though haven't really found anything detailed/complete on what would apply to my situation.

So, I thought I would give kern.hz=100 a shot.

At first it seemed to make a difference....no improvement in how long to complete a poll, but the load was lower. Until I realized that a service had failed to start after reboot. I had only run the rc script by hand, I hadn't tested it in a reboot situation. And, its not an rc script....it was used to be a single line in rc.local that worked on ubuntu and FreeBSD (except on one of the Ubuntu systems it results in a ton of zombie processes, so making it an init.d script that I could call restart on happened.

So, I spent quite a lot of time reworking it into what will hopefully be an accept rc script. One thing I had changed was that instead of using a pipe ('|') which was causing the process after the pipe to respawn and turn the previous process into a zombie each time the log file was rotated and "tail -F" announced the switch. And, this was while I was moving the service to FreeBSD (and management under cfengine 3.)

Though looking at my cacti graphs later....while the service had failed to start after reboot, it turned out to have been running for sometime, until I had broken it completely in trying to rc-ify the init script. Will, duh....I had cfengine set to promise that the process was running, and it had repaired that it hadn't started after the reboot.

Another thing I had done with I had init-ified the startup of this service, was I switched from using pipe ('|') to using a fifo, which addressed the respawning and zombie problem and eliminated the original reason to have an init.d script....

While the init.d script had worked on FreeBSD...it was just starting the two processes with '&' on the end then exiting. FreeBSD's rc subroutines do a bit more than that. So things weren't working. The problem was that even though I was using daemon instead of '&', so that daemon would capture the pid and make a pidfile. seems daemon wants the process it manages to be fully working before it'll detach. But, the process is blocked until there's a sink on the other end of the fifo. (does sink fit was the name for the fifo's reader?) I first wonder if I could just flip the two around, but I suspect starting the read process first would be just as blocked until the write process is started. So, I cheated by doing a prestart of the writing process and only tracking the reading process.

Though it took a bit more work to get the 'status' action to work....eventually found I needed to define 'interpreter' since the reading process is a perl script. And, the check_pidfile does more than just check to see if there's a process at the pid, but that its the right process. And, it distinguishes between arg0 and the rest.

Pretty slick...guess I need to do a more thorough reading of the various FreeBSD handbooks, etc. Of course, it has been 13+ years between when I first played with FreeBSD to its take over of my life now.

As for the tuning....it had made a small difference, but no improvement on cacti system stats. Basically the load average fluctuates a bit more and the CPU utilization seems to be a bit lower...though it could because the 4 lines of the cacti graph aren't so close to each other now.

Meanwhile...I noticed that one of the block rules in my firewall had a much higher count than I would expect, so I think I was about to get logging configured to see what that's about.....(which I was working on when I remembered that I hadn't rebooted after making the kern.hz change to /boot/loader.conf yesterday...the commit also picked up files that I had touched while working on moving the one remaining application on 'box', though that may get delayed to another weekend....perhaps the 4 day one coming up.)

I had set cf-execd's schedule to be really infrequent (3 times an hour), because I was doing a lot of testing and cf-agent collisions are messy....messier than they were in cfengine 2 (in 2 it usually just failed to connect and aborted, in 3 it would keep trying and splatter bits and pieces everywhere....which is bad when there are parts using single copy nirvana. resulting in services getting less specific configs, until the next run.

But, I sort of brought back dynamic bundle sequences.... but key off of "from_cfexecd", so I can test my new promise with less problems of colliding with established promises. Though there are other areas where things still get messy.... need to clean up some of the promises I had based on how things were done at work, so that the promises are more standalone.

Kind of weird using my home cfengine 3 setup, and other admin activities, as the means to break the bad habits I had picked up at work....

04/28/13

  10:06:00 pm, by The Dreamer   , 989 words  
Categories: Hardware, Software, Computer, Ubuntu, FreeBSD, CFEngine

Took a diversion from cacti and now its nagios

So, doing cacti on cbox doesn't seem to be working long term... but, the moment is being prepared for....I starting to assemble the pieces to build a new machine to do this and handle some other tasks that I've been looking for a place for.

Back to cfengine, I added a promise for dnetc (distributed.net)....and then a promise to finally configure CUPS on the two servers. And, then I turned to nagios.

I spent a couple evenings creating the initial configuration of nagios, working in design changes that I wanted to make and initial monitoring of localhost (dbox). Though it wasn't straight forward....there were differences here and there....mostly in FreeBSD layout, paths, and some of the commands taking different options. But, eventually I got everything running. My old check_dyndns worked once, but then stopped working.... problem was that it did 'stat -c "%Y" ..." which doesn't work on FreeBSD, 'stat -f "%m" ...' was the adjustment for that. All, while all the checks_* seem to be there, command definitions was lacking....but I guess having command definitions for everything is part of the debian/ubuntu packaging. There were other frills that came with that, that I don't mind not having...

I did run into check_ntp being deprecated....with check_ntp_time and check_ntp_peer being the tests to use....separating and making more clear on whether you're comparing time between servers using ntp or checking the state of the ntp server...
It did show some interesting oddities in holding NTP time on my home network.... I know that I should have 3 or more ntp servers, but it seems that I'm often landing in the state where I only have 2....with lots of delay, resulting in pretty good swings of jitter....almost makes me wonder if this something I could graph in cacti.... :hmm:

Wonder if I can find a cheap NTP appliance somewhere....

The last stumbling block was check_dhcp. Which seems to be broken on FreeBSD. All, the discussion on it seemed to point to firewalls, but no firewalls and it still didn't work....tcpdump on both places, and its saying it sending stuff, but no packets appearing on the network. But, I can see the other DHCP traffic on the network.

I remove that check and call it a night. I mull some possible work arounds....first one I tried was setting up linux compability and try running the check_dhcp from my working (ubuntu) nagios. Well, it didn't work...it couldn't find an interface. Oh well, guess there's the ugly way....use nrpe to invoke it. Though that didn't work right away.....probably because while I had created new nrpe configs for all my servers in cfengine, I haven't put any of my ubuntu servers under cfengine yet. Most of the other promises haven't been implemented for ubuntu yet. It was pretty simple to include nrpe.cfg for everything.... in fact it condensed to only 3 files.... a freebsd version, an ubuntu version and a host specific version for orac. Well, not right away...that happened more recently...while I was going through and updating the nrpe.cfg's by hand on the ubuntu servers. Was when I noticed that some of the files were only different in comments....so I made further simplifications in cfengine...which'll propagate out eventually....

Long term, I'll probably just have to track down some alternate implementation of check_dhcp....

I then add cbox to monitoring...and then looked to see about monitoring things that are on cbox/dbox...so I found checks for freeradius, cups, squid, along with improvements to checks on ntp. The check_squid was tricky....I got it working by hand, after making the suggested change for the default Cache type parsing, which turned out to be changes for squid3 vs. squid2 (but box is still running squid 2.7 - since I had re-built it by hand with SSL support, and blocked ubuntu from updating it. Orac wasn't blocked so it eventually turned into squid3.

it worked by hand, but wouldn't work under nagios...turned out that the embedded perl wasn't liking it. I was going to disable embedded perl for it, when I took a look at seeing what it was complaining about. And, did some reading on embedded perl.... the gist was "use strict", "perl -w" and "perl -c" as starting points. perl -w was find, but perl -c had one problem....which I fixed. But, no go. And, then noticed the line "# todo : use strict", guess I'll have to deal with that.

And, making that all happy, got it working.

The only other quirk was the memory check wouldn't work on FreeBSD, I guess there's no mallinfo() available for that. So, no running that test on those servers....plus no Cache test on box. But, it still left enough variety of tests that worked on all. And, it wasn't so much that I wanted to get all the information, but I choose to define all the different tests with ports set into the test....so running the check would also test that all my squid ports worked. There's actually only two that matter, but I have all my squid's configured the same, listening on 5 or 7 ports....depending on whether I have SSL enabled. Though I pretty much only need two now. I'm not doing transparent proxying and I don't need the SSL now that I've split box into dbox/cbox....the SSL was so ddclient could work on box and update dyndns via proxy to DSL....

Next up is adding zen to nagios, and coming with with more tests of things that are specific to zen, but covered or not covered in the old nagios.

Though as I worked along...there were things I couldn't find monitors for...though I realized that I could have cfengine promise that those services were running. Plus cfengine was also taking care of other things. So, I should probably work on writing some promises for zen. So, I can have promises to make sure things are started up again after a port is updated or that php/extensions.ini is reordered, etc.

But, I'll probably continue adding everything else to nagios first.

04/21/13

  10:52:00 pm, by The Dreamer   , 2431 words  
Categories: Software, Computer, Ubuntu, FreeBSD, CFEngine

Home server migration ran into some cacti

The home server migration that I wrote about on April 7th, hit a delay .... I started working on migrating cacti and nagios.

I probably should've started with nagios, since I don't think that would've taken as long as cacti has.

I had already been monitoring the new servers using my old cacti installation. I had pretty much decided that moving the old installation to the new servers wasn't going to straightforward.... partly because of versions, and no easy intermediary. But, I wasn't too worried about the historical data in my old cacti....

I figured that once I got things up and running, I'd just export the templates and import them into my new system and I'd be done.

But, then I hit a hitch....the squid templates I had weren't working on the new system....all I could find were old results about issues with doing SNMP to ports other than 161, and possibly due to newer versions of net-snmp....though that later turned out to be a wild goose.

Anyways...the work around was to use the proxy option in net-snmp. Though I recall having tried net-snmp before discovering bsnmpd on FreeBSD, but I gave it a shot.

Before I got to testing the proxy...I soon saw that it wasn't giving the same information as bsnmpd...specifically, for the HOST-RESOURCES-MIB and parts of UCB-SNMP-MIB. So, I decided that I could proxy net-snmp to bsnmpd and get those. But, that didn't work.....after some reading the answer was I needed to either map bsnmpd in somewhere else or exclude those areas from net-snmp.

Well, during the build of net-snmp, it did make reference to being able to set some variables in make.conf -- such as NET_SNMP_WITH_MIB_MODULE_LIST and NET_SNMP_WITHOUT_MIB_MODULE_LIST. And, by default NET_SNMP_WITH_MIB_MODULE_LIST contained "host disman/event-mib smux mibII/mta_sendmail mitII/tcpTable ucd-snmp/diskio sctp-mib if-mib"

So, I tried setting NET_SNMP_WITH_MIB_MODULE_LIST without host and ucb-snmp/diskio and tried to exclude the rest of ucb-snmp in NET_SNMP_WITHOUT_MIB_MODULE_LIST. Which got me a strange error about host being in both lists.

I delved into the Makefile, and found while the other settable NET_SNMP parameters were done as '?=' in the Makefile, the NET_SNMP_WITH_MODULE_LIST was done as '+='...with conditionals that '+=' the last two modules.

OSVERSION >= 700028 adds 'sctp-mib' and the port option MFD_REWRITES adds 'if-mib'....I had started looking at what the fix might be, but decided that all I needed to do was remove all these lines...since I'm going to have my own definition in my /etc/make.conf file.

Trying to exclude all of ucd-snmp wouldn't make things work....but I did an snmpwalk comparing bsnmpd and net-snmp, and decided that the two areas that were lacking were ucd-snmp/diskio and ucd-snmp/disk_hw. So, I recreated the 'original' NET_SNMP_WITH_MODULE_LIST in /etc/make.conf, without 'host' and 'ucd-snmp/diskio' and put 'ucd-snmp/disk_hw' in NET_SNMP_WITHOUT_MODULE_LIST. The build grumbled, but finished.

I that worked.....all my ucd/snmp host graphs were working on m new cacti server in the same detail that I was getting before (IE: the CPU Utilization gave traces for each of the 8 vCPUs...instead of just one.... I could see all the ZFS filesystems, not just the the single zroot.

So, I went back to looking at getting squid graphs to work....that didn't work.

Pages: 1· 2· 3

04/18/13

  12:47:00 pm, by The Dreamer   , 1259 words  
Categories: Software, Networking, AT&T DSL, Broadband, CFEngine

Zoom ADSL X3 5760 & Cacti

It was a dark and stormy...late afternoon...yesterday, and....

I had started out almost 7 years ago with a Siemens 4100 DSL Modem, which worked the way I needed it to for my home network. And, wasn't sure how easy it would be to find another like it. I was running it in the cross between router and bridge mode...so that my router could maintain my dyndns info (though it wasn't too long after that I moved that to ddclient on box, which has been more reliable...but I was having ddclient scrape from the router, though the ddclient for the router on my Cox connection wasn't supported so that uses checkip.dyndns.org. So, now both do.

Would probably be too much work to make ddclient go out on the right IP so that ip route will send it to the DSL router, so it can query the DSL modem for what the real external IP is. Though the new cbox/dbox setup would simplify things....but the migration has stalled as I've been working on getting cacti moved from box...and it hasn't been going well. Lots of old templates and such don't work on the new, so I've been reworking what I feel I can't live without....

That includes the graphs of my DSL modem stats....

Anyways....when the Siemens 4100 started dropping the connection a lot (around the 3 year mark) and changing the filter didn't help, I had heard that these things wear out... So, I tracked down a new Siemens 4100 on eBay...and switched to that....and that got things working again.... Then a couple years ago, things go bad consistently....though I could see from my cacti graphs that SNR drops in the evening. Though I wasn't able to get local service to restore/fix things. I tried the AT&T forum on dslreports.com, and they changed me to Interleaved, which helped....

But, I had started shopping around for a new DSL modem.... somewhere in my journey's I acquired a Zoom ADSL X3 5760 Modem. But, since things were working...I put it aside as my spare for when things stop. Seems I've had it so long that its no longer available....got it July 9, 2012 according to Amazon.com

For a while now, it would drop the connection now and then during the week (between its weekly self-reboot)...at first I suspected the router, since its twin had gone away in much the same way several months earlier. Though the router do also have failsafe configured, so if it can't talk (ping) to box or the WAN gateway...it reboots. Though at some point AT&T made their gateways unpingable. So, it was pinging google.

But, on April 6 it got really bad....my IRC connection was resetting practically constantly. Though since I had swapped the router before, and swapped it again. Though maybe now I wonder if its watchdog was too aggressive. Things were usable, but the line drops would be annoying. Also the IP staying the same through drops didn't make me question the DSL modem.

But, then on April 13, things start getting really bad....and I was getting 50+ messages a day from ddclient that my IP changed. It seemed to stablize a bit on Monday....though it was still dropping regular enough that I switched to using Cox for my IRC screen session. Was going to defer to the weekend to make the swap.

Well, yesterday the weather was bad...lots of lightning, rain....and I first display I looked at when I got home said "NO INTERNET". Though it was probably a temporary outage, because it did appear to eventually come back while I was working on unboxing my 'new' DSL modem. And, try to figure out how to set it up without the Windows wizard it provides or the lack of documentation with it...there was a small CD, which didn't really provide much depth....but I found what IP it would be and that it has web interface....it also has a telnet interface and an FTP interface.

Anyways...it turned out to be pretty straight forward getting it working...the hard part was figuring out what the non-default options meant, and whether I would want them.... the main one I turned on was "fullcone NAT". And, I set my router in with a reserved IP and made it DMZ host, so I can keep all my forwards there...plus the Zoom is limited to 16, which isn't enough .... though this may change when I make use of its DMZ feature as well (doing reverse proxy on cbox/dbox to everywhere else on my home network...running firewall on these boxes already, to implement policy based routing.) And, enabling ICMP on the WAN interface (its also possible to enable http, ftp and telnet on the WAN interface as well.)

Getting it working in Cacti again, turned out to be much harder.

Full story »

Pages: 1· 2

04/07/13

  11:29:00 pm, by The Dreamer   , 2462 words  
Categories: Hardware, Software, Computer, Networking, Operating Systems, Ubuntu, FreeBSD, CFEngine

The home servers migration got off to a rough start...

My 10.04LTS servers are nearing the end of the line, but the move to get them upgraded is starting to get close....

At first I was waiting for the 12.04.1LTS update to appear, so that I might try upgrading them to Precise. But, then it struck me that it struck me that it might be time to replace one of the servers with new hardware.

The boxes had been servers for many years now, but in 10.04 they dropped support for 32-bit hardware as servers....which means one of my servers, and the more important one...is nearing EOL.

So, back in August....I came across a Shuttle XS36V open box on newegg.com....so I ordered one, along with an 8GB SODIMM memory set (a pair of 4GB)....the computers support a maximum of 4GB, but in my mind I was thinking that if this worked, I might get another to make a pair.

But, then it sat around as I go on to other projects....I had figured on using one of the 120GB SSDs that I had acquired earlier....

Realized what was strange about these boxes was they had serial ports, so I started thinking about network serial and setting these boxes up headless. Since I had done a server at work for serial with FreeBSD, I got to thinking that I would do the same with this. Plus I was really finding to be much more predictable/stable than Ubuntu....something that's good in a server.

I started revising the plans as I went....ended up deciding that I would have one that is default cable and one that is default dsl, and running the usual core...one would be master DNS, and the other one be the primary slave....just as one would be primary DHCP and the other secondary....both would do freeradius. Both would NTP servers. Both would do mail, though zen is my main mail system. They would have both apache and nginx on them...apache to run local web services....and nginx doing reverse proxy to the other web apps on my home network (kind of important since current cable router is a step back with it only supporting 16 port forwards....though I have a replacement that I'll get to setting up some day...)

Then while I was poking around on newegg.com, spotted that there was another open box Shuttle XS36V....so now I had two boxes. I had picked up once Cyclades ACS1 earlier on ebay, so I got a second....a 'new' one for about the same price. Just needed to find time. Perhaps after FreeBSD 9.1 drops.

Well, it dropped....but I still wasn't ready...now I needed some SSD drives for the machines. So, I used the $30 in RewardZone for the new TV I had gotten in September, and picked up a pair for 128GB SanDisk Extremes. And, then I started setting up the ACS1s....hmmm, wonder if I have any serial cables. Okay, I'll buy some of those first.... I end up ordering from TigerDirect, as they have a better price on RJ45 to DB9 adapters....not that I need them for the actual ACS1 to Shuttle XS36v connection...that's just a 9F to 9F null modem cable.

Pages: 1· 2· 3· 4

11/25/12

  07:55:00 pm, by The Dreamer   , 1327 words  
Categories: Software, Computer, Networking, Operating Systems, FreeBSD, CFEngine

VLANs & FreeBSD 9.0/Jails

The backstory is that I've been working in a primarily Sun shop, and one of the things we've been doing is running Solaris 10 on large boxes, such as T2000's, T5120's, M4000's and cutting them up with Solaris zones/containers with the global native into a management vlan and tagging appropriate vlans for the zones, and the zones have their own default route specification so all has been great.

ipf on the global so the zones can't tamper with their own firewalls, and on some of the 'zone' servers using:

/usr/sbin/ndd -set /dev/ip ip_restrict_interzone_loopback 1

The zones remain isolated from each other. Or it avoids problems of short circuiting (asymetric routing + wrong IP)....because some of the zones are behind the F5.

For years, we've tossed around the idea of introducing FreeBSD into our datacenter, and finally one of our customers decided that while they really like the containers/zones and ZFS, the cost of replacing their aging Sun server would be better done by replacing it with a FreeBSD server. While they own the hardware, and it resides at their facility...we provide the system administration support. But, this opened the door to having FreeBSD on our work site. I was in the process of replacing my aging Sun Ultra 20 with an Optiplex 990...which originally I was looking at install Ubuntu on, but instead I went with FreeBSD 9.0 (though the effort in getting the Desktop working on it, and recreating my Sun desktop/work environment on it...made me question if that was really the right way to go. But, I got it working.) And, it helps me try some of the things before doing them on the customer's server (which is headless in a closet [well, they have a monitor & keyboard for console access], so all the desktop stuff was for my benefit...it paved the way to me getting a working FreeBSD desktop at home &#59;D .... though I may end up with a different system for my main desktop at home and have the FreeBSD machine go headless with my other FreeBSD servers....unless there's some way to easily share between the two....switching doesn't qualify.)

So, in needing to deploy some new internal services (such as monitoring) and not really wanting to go through the major process of find all the bits and pieces and creating packages under our CM system for Solaris. Its quite the pain building each and every perl module as separate CM packages, instead of having some system that automatically builds and installs (or makes packages) for you...ala ports or CPAN. I've done package install requests that start out as install one package, and end up building 100 or so packages instead.

I had contemplated sneaking Ubuntu in since I run the same monitoring servers on an Ubuntu server at home, but the work to incorporate Ubuntu into our configuration management infrastructure got sidelined by FreeBSD. And, there's no decision on whether Ubuntu will come into play (though the high cost of RedHat licenses to just get patches...for systems that are rarely patched....is making Ubuntu look attractive.)

Anyways things led to me starting work on pxe boot installing Proliant DL380s with FreeBSD 9.0 and creating 'jail' servers to work like our 'zone' servers.

Cloning interfaces to do VLANs was simple ....

Full story »

Pages: 1· 2

2

Now instead of subjecting some poor random forum to a long rambling thought, I will try to consolidate those things into this blog where they can be more easily ignored profess to be collected thoughts from my mind.

Latest Poopli Updaters -- http://lkc.me/poop

bloglovin

There are 20 years 3 months 25 days 22 hours 33 minutes and 52 seconds until the end of time.
And, it has been 4 years 9 months 1 day 15 hours 29 minutes and 4 seconds since The Doctor saved us all from the end of the World!

Search

September 2017
Mon Tue Wed Thu Fri Sat Sun
 << <   > >>
        1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30  
Google

Linkblog

  XML Feeds

Who's Online?

  • Guest Users: 1
This seal is issued to lawrencechen.net by StopTheHacker Inc.
free blog tool

hosted by
Green Web Hosting! This site hosted by DreamHost.

monitored by
Monitored by eXternalTest
SiteUptime Web Site Monitoring Service
website uptime