development and infrastructure29 Jan 2013 03:47 pm

I have been playing with an HP 1U server for the last few days, installing CentOS 6.3 on the server, and getting it ready to be staged in the Data Center as part of the Data Loss Prevention (DLP) system QA environment.

I always recommend that when setting up any type of computer system for a company you first build out a full sized QA environment. That way you can always test upgrades or troubleshoot problems in the QA environment without impacting your production system.

Its interesting that some companies don’t want to expend the extra cost of putting in a QA system, but, now a days I insist on having a full QA environment. Without having one, nothing good will come of it.

A best practice that I learned from the company I work at is very interesting. NOBODY goes into the data center during daytime business hours. And, after hours, the only people that can access the data center are people that work for the Data Center operations team. If you need access to the server, you can to use the Lights Out access program.

This best practice reminds me of a funny story for a company I was working for MANY years ago. My cube was outside of their main data center. One afternoon, there were two guys working on the electrical panel outside of the data center that had the main cut off switch for all power in the data center and the building. These two guys had a ladder, and took off the front door of the electrical panel, and for what seemed like a good idea at the time, they hung this door above the panel.

So there I was sitting in my cube, and all of a sudden, I hear a POP sound, there is PITCH BLACKNESS in the room, followed by a few clangs and all of the power goes out. My computer, the lights, everything in the building. And then, a few seconds later, emergency lighting goes on.

The panel they they perched above the main cutoff switch fell from its perch, and hit the breaker, causing all power to be cut to the data center. And because that switch was thrown, the transfer switch did not transfer power to the batteries and generator and all power in the data center to all of their computers was shut off, causing their mainframe to shut down hard, which disrupted the order entry system for all of their 375 locations around the country.

After a hard shutdown, it took them 45 minutes to bring the mainframe back up after they two electrical works debated for a few minutes about flipping the breaker back restoring power to the facility. So I can understand first hand why NOBODY goes into the data center during business hours, and after hours, only the data center operations team.

Which brings me to the part of the blog entry I wanted to blog about. Using HP Lights Out, from my desk, I can access my server as if I am sitting in front of it. I needed to do this today, because when setting up the network for the server, I must have fat fingered the default network route so I was unable to SSH into the server until I fixed the default route today.

I was telling my son yesterday that we are very fortunate today to have Google at our fingertips with manuals and instructions for virtually everything available, including how to restart your network in CentOS when you change your default route. In the old days, you had to memorize all of the useless trivia facts and esoteric UNIX commands instead of being able to look them up on the fly.

So, my CentOS server is now installed in the data center, and security patched using YUM, and is ready to have the DLP component installed on it later today and added to the DLP QA environment. Another fun day.

development and home and infrastructure28 Dec 2012 07:31 am

Time sure flies in the blogsphere as I noticed that I have not blogged about anything in about a year.  Not to say that the conversation is not occurring.  It occurs in real time through Facebook and Twitter, with thoughts or ideas that internet me getting a status on Facebook or a tweet on twitter and then, my friends and colleagues interactively discussing the topic with me.

My home office PC was having all sorts of strange problems ever since I had a power failure at my house that took out one phase of our three phase power.  Even though the computer was connected to a UPS and we have a whole house generator, the one phase being down did not cause the transfer switch to move us off grid power, and the brown out went right through my UPS, and my quad core PC started frying.  I replaced the power supply, and the video card, but, that computer was still completely screwed up.  The behavior is very strange.  At no particular interval, processes freeze on the computer. In the task manager, they are listed as “not responding”.  I reinstalled the operating system twice and it didn’t solve the issue.  So I decided to get the iMac 27.   This was one of the best computer decisions I have made in my professional career.  Its remarkable that a new computer can fill an lifetime IT guy with Glee.  After I made the decision about 6 months ago to get the new iMac, I decided to wait until Apple released the refreshed next generation iMac.  And that was also a fine decision.

The computer itself by its very design is far and beyond that of a typical Windows based PC.  While you can get these options for a Windows PC, by default you get a small wireless keyboard that has an excellent feel, a trackpad to use as a mouse and all of the technology built into the 27 inch display.

The setup process was a breeze, getting the machine on my home wireless network, and connected to the physical network.  The access point in my home office is older so its not using the latest and greatest wireless protocol so the speed is better on the wired network.  Unlike the new access point i installed downstairs near where the boys Xboxes are located.  THEY are enjoying 56MB connectivity from both of their Xboxes.

My home office computer really has only two major functions at this point.  One is using the Microsoft Office suite of software and the other quickbooks.  Microsoft Office 2011 for the MAC is great.,  And Quickbooks Mac is also an excellent release.

When I got my MacBook a few years back the genius at the Apple store recommended Parallels VM software for the mac.  It has worked out great, letting me run Windows XP so I could use Windows Media Player to watch the hundreds of movies and such I have collected over the years.  I also set up a Windows 8 VM to check it out, and I still think that the new Windows look and feel is strange.  They tip their hat to the surface tablet a bit too much for my taste for a desktop operating system.   This again smacks of Apple really understanding what a end user wants and needs to do with their computer.

And, I’m in the process of downloading the latest CentOS Linux operating systems which I’m going to slap into a new VM on my iMac.  I need to set up that operating system on a 1U server for the project I am working on at work.  I was using a shared Linux server for my QA environment for the project I am working on and was kindly asked to get off the machine since they needed it for something else.  When I requested a new Linux VM to the UNIX team, they told me that the request would cost me a charge back of $11,000 for the VM, but, if I wanted a physical 1U box, I could have that for free.   $11,000 versus free.  The free wins out.   Thats another reason why my vast experience in all aspects of IT pays off again, as I can install the operating system and stage the computer myself at my client instead of having to engage other people to help get the computer set up.

And having thew ability to throw down a Linux VM on this iMac is cool because I can test the install before going into the data center lab to put the OS on the 1U server.

So I am completely pleased with the new iMac. And will be continuing to copy over pictures and other media from my old quadcore to the 3TB hard disk on this iMac.  I was also going to look for a NAS for the house that has a RAID 1 hard disk that we can all use to back up our files at the house.

Merry Christmas and Happy New Year to All!  Hopefully it will not be another year before i blog again!

development15 Oct 2009 08:02 am

A client who I did a project for in the past contacted me and asked if I had some availability to migrate over a system I developed from the really old infrastructure it was running on to some newer hardware.  Once migrated, my client could end support for the really old hardware, and turn it down saving money.  Great idea.  Could I provide an estimate on the amount of time required to get everying migrated and working?  Sure.   I think about it and go over what was required for the previous activity and esimate about 2 weeks or so.

The old environment was a Solaris 2.6 server.  The new environment was Solaris 10.

The new system required me to install GPG encryption and PHP to work within a SunOne iPlanet web server.  Sounds simple?  Right?  Wrong.

While Sun powered the Internet boom around the late 90’s and year 2000’s, Sun pretty much fell by the wayside.  Sun purchased Netscape iPlanet, and bundled the web server with their standard distribution.  But, the Open Source movement and the Free Software Foundation came out with Apache.  A web server that performed circles around SunOne iPlanet.  My friend Ron did some really good performance testing of SunOne, and could demonstrate chapter and verse how an Apache web server could support way more users then an iPlanet web server.

Many high performance web sites still running on Solaris eventually migrated over to Apache to get the more bang for the buck.  But, for web sites that were not highly scaled, why spend the dollars to migration from SunOne iPlanet to Apache if everything is working?

The problem comes in when you attempt to find software distributions from Sun Freeware that are packaged as executables for SunOne iPlanet on Solaris 10.  The only pre-compiled packages of PHP were for Apache.  The source code was available, but, to build PHP, there were many many MANY steps.  I’ll rattle off a few of them.

– Install lots of packages


– Libgcc

– A bunch of other libraries

– gmake

Then I find out that the Solaris 10 development server I was using was installed as a development server.  When you install Solaris on a server, you are asked which flavor you wish to install.  Production (stripped down), development (more stuff), or Full Distribution (everything).  Trying to get GCC to complile things required MANY libraries be installed on the server.  Again, more time required to get everything working.  And, while trying to debug while your make and compile was working, you get really descriptive errors, like “Error 10” – Compilation failed

Error 10!  What’s error 10?  google and google until I find a roadmap down the pieces and parts that are not working.

Finally, everything compiles, the software is installed, and the web site looks to be working, then need to tweet the GPG encyption so it uses the previously generated keys.   Then a bunch of debugging on the PHP to encryption component.

Everything finally started working, but, unfortunately, it took about 3 times longer then I anticipated.   Thats the problem in IT when you attempt to estimate a development cycle in a vacum with on partial information avaialble.

The project was fun because I really had to really roll up my sleves and get into a development cycle.  

The client should recoop the investment by turning down the old infrastrucutre, which rarely happens in IT.

development and infrastructure04 Aug 2009 05:43 pm

A client I worked for a few years ago called me and asked if I had some availability to move PHP, GPG, and some web pages and PHP from a legacy Solaris 7 box running Netscape web server to a Solaris 10 server running Sun One Java Web server.

Of course, I say, since I like doing hands on technical work, and I always like to help out my clients.  They were interested in moving a bunch of web sites from legacy hardware in the data center to current technology to lower the TCO by spinning down the old boxes once the migration was complete.   Sounds like a good plan.

The best laid plans of mice and men begain when I went to a Sun Software repository web site and tried to find a compiled PHP library for Solaris 10 and Sun One web server.  Suprise Surprise much of the world has migrated away from Sun One web server to Apache since Apache is free and has better performance.  I’ve always been a Netscape Sun One guy, but, a developer that I respect did some performance benchmarks of Sun One webserver versus Apache and the differe in cycles were staggering.  You could support way more users on Apache then Sun One on the same class Solaris server.  That explains why much of the world has migrated to apache.

Back to my story.  No compiled library for PHP and Sun One and the NSAPI could be found, so it became software development 101 time.  I needed to build the PHP library from the source code.

Sounds simple.  Only I install the GCC package, and the compiler barfs about missing librarys.  Googling and noodling remind me that when installing solaris you could pick the base install.   Production server, database server, development server, etc.  And based on your selection, different libraries are installed on the server.

Not that GCC was being helpful.  It barfed up fur balls about missing libraries without much information.

After getting GCC going, PHP needed about a dozen other things installed before it would compile down to an executable.

The same trial and error process occured with the GPG library which provides encyption at no cost compared to the PGP solution.

Finally I got everything compiled and installed.  PHP for  NSAPI, GPG, the forms and all of the scripts and bingo, everything worked like clockwork.

It was really fun rolling up my sleeves and getting technical.  And I was happy to help out my client.

development29 Jul 2008 05:39 am

I placed an ad on a couple of Internet sites looking for a Microsoft Vista PC Image build expert for my client who needs to upgrade 21,000 desktops.    Next came the steam of emails from job applicants looking for a job.   I’m not 100% sure that I got the virus from one of those emails, but I will describe the behavior I noticed.

When I attempted to open a word document, I got a message that my Norton anti-virus was having trouble and could not open the email to scan it.   Something had shut down my Norton anti-virus, or at least caused to to quirk out  I was not particularly worried, but, I figured to fix the problem when I had a bit of time.

Flash forward a day later, I get home from work, and my wife tells me she can not send email.  I check my computer and sure enough, it can’t send emai either.  Time to figure out whats going on.   I check my Norton anti-virus on my computer, which tells me that it is hopelessly screwed up, and I need to completely uninstall and re-install it.    Now what could cause Norton to blow up on my computer I still have no clue.  But, I proceed to uninstall it.   After that process, Norton informs me that it needs to reboot.  I tell it to do it, and it starts shutting down everything on my computer.  Right before it reboots, word pops up on my screen, with an action box, that has a long string of asian characters, and an OK button.   Very strange.

I reboot the computer, download a fresh copy of Norton antivirus and go through the install process.  I then scan the computer, do a live update, and scan the computer again.   Norton sees no sign of a virus.

I called Comcast twice the night before to ask why SMTP outbound was blocked, and the level 1 support person had no clue what I was talking about.   When I called this morning, and explained the issue as, I may have had a computer virus that was sending email, and now I can’t send SMTP, the tech this morning said I would need to call the Abuse/Legal department when they opened later today.  It was very silly that when they did block outbound SMTP I didn’t get a notification, a call, an email, or at least they update their system so the techs you call for support know that it was blocked.

I still don’t know what virus or spambot was running on my machine to generate SPAM and cause Comcast to block SMTP outbound.   I scanned the computer twice,  and live updated a couple of times.  It will be interesting to find out as I continue to chase this down if it was a transient virus that left no trace when the computer was rebooted and the antivirus software uninstalled and re-installed.

The fun will continue this morning when I speak with the Comcast legal/abuse department about turning back on my ability to send SMTP email.

development27 Nov 2007 02:03 pm

I received a request from my client to re-code a legacy application that performed audit activities on the software system that I am supporting.  The source code for the legacy application was not available, so I needed to start from scratch.

The software program needed to run on Windows PC’s, so Microsoft Visual Studio was an excellent choice for developing the software.  Back in the day, I had been a UNIX “C/C++” Developer, a Microsoft VC developer, a BASIC programmer, and I have dabbled with Java.”, so creating a new application should be a piece of cake.

Microsoft Visual Studio is VERY cool.  You can develop your application in ANY language that you like, and access ANY of the .NET framework libraries.   You can code in Java, C++, or VB, with complete access to all of the .NET framework tools.   This is absolutely amazing.

I rewrote a critical legacy application that took over an hour to run, in just about 3 hours of coding time, using many of the cool functions available in the .Net framework, and found that the application completes with perfect output in just under 1 second.  My client was blown alway by the new application, the speed, and the expanded functionality.

After leading the ENTIRE enterprise security department in a CISO role, with SOX compliance accountability, it is really fun to get back down to development rolling up my slieves, and solving business problems.   Its nice to know that I can still go back to my roots and code software.

Kuddos to Microsoft on a well written development suite.

development11 Oct 2007 04:40 pm

I think that some of my development buddies would get a kick out of some of the work I have been doing to support my current client.  As I shared earlier, they had a key developer walk out right before some significant changes needed to be made to a legacy computer application that supports their business.  They brought me in to make the changes to keep the business running smoothly.  The thing is, the old software does NOT have a visual development environment.  In fact, if you attempt to edit the files using the Microsoft development environment, it munges the files so that they will no longer compile and link.  And forget about any type of iterative debugger.   Command line compiles and no debugging.  TRUELY primative.  And lets not forget the Glory of figuring out spagetti code.   And figuring out how to get the actual software to compile was another treat.  But, the more I look into the software suite, its a very Elegant solution given when the software was developed.

Working in this environment reminds me of some of the early development work I did on Sun/3 computers using the Original Glockenspiel C++ Complier.  It was not really a compiler but more a C++ pre-processor that took your C++ code and converted it into “C” and then used the standard Sun “C” compiler.   It was virtually impossible to debug anything, since the C++ code was complete tokenized and stripped before being compiled by the “C” compilor for speed.

One of the development tricks I used back then was to compile constantly to see where syntax errors occurred.   This process allowed me to unit test the code so I was very confident that the code was solid.   I’ve been using the exact same process with the development work at this client to ensure that my code changes work, with the trick being not breaking any down stream functionality.

Getting back into a development role for this project has been a real treat.  I sit in a room with another consultant and a PM for the company, and its really fun shooting the breeze during the day, eating pastacio’s from Costco, and solving business problems.  In recent years, I spent alot of time in the “C” suite as a CISO for a multi-billion dollar company directing 50+ people as well as a high level enterprise security consultant.  Its nice to know that when push comes to shove, I can roll up my sleves and solve business problems.