Ouch...server dead for almost 24 hours

Da Nag

Da Nag

Administrator
Staff member
We're back. Sorry, folks...my bad.

What was intended to me a minor update yesterday, quickly turned into severe ugliness. A patch applied to the server required a reboot, which normally takes a minute or so.

The server failed to come back up...which is an absolute PITA to deal with remotely, as I'm not sitting in front of the computer to see what is going on. My only recourse, was to deal with two levels of tech support, playing a long and frustrating "game" of telephone.

Support is by email only, and to make matters worse, my support contact doesn't do the troubleshooting in front of the computer. Instead, it went something like this:

Me: "Please try this."

Wait 5 minutes to 2 hours...

Support: "I'll forward that to the data center techs."

Wait 10-30 minutes...

Support: "Didn't work. Should we rebuild the server?"

Me: "Hell no! Try this."

Repeat the above for about 24 hours...

Anyway, everything appears to be functioning again, and lessons were learned from this experience. In particular, I put together documentation for the hosting provider, that allows them to bring the server up via a CD when the server won't boot on its own. In this "rescue mode", the server doesn't function normally - but it does allow me to get onto it remotely, and fix things directly without the game of telephone mentioned above...

Welcome back, friends...
 
Da Nag":d7qdolzs said:
What was intended to me a minor update yesterday, quickly turned into severe ugliness. A patch applied to the server required a reboot, which normally takes a minute or so.

The server failed to come back up...which is an absolute PITA to deal with remotely, as I'm not sitting in front of the computer to see what is going on. My only recourse, was to deal with two levels of tech support, playing a long and frustrating "game" of telephone.

Support is by email only, and to make matters worse, my support contact doesn't do the troubleshooting in front of the computer.
Click to expand...

Bill -- been there, done that -- but fortunately hands-on. Trying to anticipate all contingencies when writing up instructions can be very frustrating. Thanks for your efforts!

What was the date and time of the backup that you restored from? That will let us know how much activity was lost.

Warren
 
I didn't even understand 10% !

On the otherhand, what I DID understand was that I was beginning to shake from Brat Withdrawl; it was not a pretty sight!

Glad we're reconnected with the Brat Community!

THANK YOU, THANK YOU, THANK YOU, THANK YOU, THANK YOU!!!!!

Best,
Casey
 
Doryman":2fmutq2h said:
What was the date and time of the backup that you restored from? That will let us know how much activity was lost.
Click to expand...

Didn't have to go to backup - there was zero data loss. Once I had access to the server, I was able to repair the reason it wasn't coming back up on its own.

For the nerds in the audience...the ultimate problem, was a grub failure. In my best attempt at laymen's terms - the code that tells the server how to start, became corrupted by the patch I applied. It didn't affect any of the data on the server - just the startup process. Once repaired, the server came back up unaffected by the problem.
 
DaNag................

You are just great at keeping us "fat fingered" C-Brat junkies up and running. Thanks for your expertise and your effort in doing that for us. I am only a 5% understander. I also suffered withdrawal pains but am over it now.

Jack in Alaska
 
Thanks Bill. We sometimes need something like this to help us remember all the hard work you and Mike do to provide this great site...and how important this site is to us.

Thank you!
 
Losing c-brats.com is a little like the shock of losing electricty. Wow. Glad that's over. Had to click on "Donate Now" for the effort that y'all put in . . .
 
Thanks much for the kind words.

But, be careful with that Tip Jar. I just might realize, that screwing things up around here can be a money maker. Heck...maybe next time, I'll take a week to fix things - just might get enough for a TomCat. :shock:
 
Da Nag":lhn1lzhh said:
Thanks much for the kind words.

But, be careful with that Tip Jar. I just might realize, that screwing things up around here can be a money maker. Heck...maybe next time, I'll take a week to fix things - just might get enough for a TomCat. :shock:
Click to expand...

OTOH, given enough tips, you could afford to host the site where you have hands-on access!

Warren
 
Da Nag":3v66pkl6 said:
Thanks much for the kind words.

But, be careful with that Tip Jar. I just might realize, that screwing things up around here can be a money maker. Heck...maybe next time, I'll take a week to fix things - just might get enough for a TomCat. :shock:
Click to expand...

sorry I am a bad tipper
 
What was intended to me a minor update yesterday, quickly turned into severe ugliness. A patch applied to the server required a reboot, which normally takes a minute or so.
Click to expand...

As a Computer Programmer for 25 years - I know the feeling, been there done that - but in my case it would take down the mainframe for 2500 employees for a bit, not good.

Very cool that c-brats have someone like Da Nag with the knowledge and the willingness to get it done.

Many thanks.
 
"the ultimate problem, was a grub failure"

I had my grub fail once.... :shock: Turns out it was one of the raw oysters I ate!! :roll:

Thanks for the hard work Bill, we missed you almost as much as we did the server!!

Charlie
 
Captains Cat":24upz8eb said:
"the ultimate problem, was a grub failure"

I had my grub fail once.... :shock: Turns out it was one of the raw oysters I ate!! :roll:

...
Charlie
Click to expand...

(Insert your own "raw oyster" joke here)

Bill, thank you for dealing with the frustration... there's no place like "home"; and that's what this place is for many of us.

I missed you folks.

Best wishes,
Jim B.
 
grub failure
Click to expand...

I had that in the lawn once and it killed everything.

Thank you for taking care of this wonderful site for us. I have just replaced my lap top due to a progressive list of failures. Can’t live with them can’t live without them.
 
Back
Top