Have you ever had one of those weeks where nothing seems to go right? This week smells and tastes like just one of those. For my buddy, Dan, it's been even worse. He's been dealing with server problems all week long.
Starting Sunday, our main images server went down. Then database replication failed, forcing us to dump and re-import to the slave. What a lovely way to start the day at 6am. Early monday morning, the same images server went down. This time, a different error. Are we dealing with hardware issues here? Maybe? Lovely.
Tuesday, same time, different error on an entirely unrelated part of the system. WTF. Bad memory? I rushed an overnighted order of 1gig of memory to be safe. Was a cron job causing all this? Dan and I try to narrow our search. It appears that updatedb may have been causing the system to freak the hell out with the sheer number of files (images). We exclude that path from the list of directories it indexes and turn off logrotate (which runs daily and ensures logs don't get too big).
I spent most of the day re-implementing a part of our current (unrelated) project because it was too cryptic and difficult to debug. What a waste.
Wednesday, nothing. No problems. I can safely focus on our main project. All around, a good, productive day even though I spent it debugging someone else's code that was obviously rushed. Maybe I'll get to work on my code finally? No, no. Of course not, silly. I continued to debug this other component of our system that I didn't write. One of the bugs ended up being memory-management related, of which, bugs which are usually quite difficult to track down.
Thursday, again -- nothing. Yay. I am relieved. Things seem stable. More debugging and development. Once the evening rolls around, we get to endure more server problems. Dan ended up with corrupt database tables after a power cycle to install power switches onto our kvm so we can reboot servers remotely. Never fun. Once things seem to be up and running, we notice bandwidth to the servers is slow and chunky at best. After a lot of head-scratching and craziness, we determine that our firewall didn't come back up with a full-duplex connection on our network drop to our cage/rack. After forcing the connection to 10baseT full-duplex (previously was 10baseT half-duplex), the site starts hauling and things speed up to their usual zippiness.
The surfing Thursday night was awesome. A huge SW swell has been building over the course of the week and Thursday was the big ramp-up. Megan and I surfed Seaside until the sun went down. The place was hoppin. I counted 34 people when we first got out there and I'm certain more showed up over the course of the session. Tons of fun.
Early this morning, we had to go out again even though my upper body was entirely spent from the past few days. It was the first time I had done a dawn-patrol session in a long, long time. It's so incredibly nice to get out there when the ocean surface is smooth and glassy. Especially with this nice SW swell holding strong and Torrey Pines is breaking just right. There were an insane number of dolphins out, too. They were surfing waves and just hanging out all around us the whole morning. They seemed to be in a very jump-up-in-the-air-happy mood and were launching in and out of waves all around us. The guy next to me had one jump up, do a flip and land not 5 feet from his board. Crazy. The baby dolphins were cute to watch, too.
Today is looking good. Everything always seems great during and after a nice surf session. Tonight is the season 2 premiere of Battlestar Galactica -- something we've all been anxiously awaiting. We're getting together at Kiyomi's place for dinner before the show. Can't wait!
