This past week or so I’ve been concentrating on improving the performance of our system. Thankfully this isn’t a solo task and my teammate really knows what he’s doing. Luckily I was able to contribute based on my past experiences with things like cache settings in Apache, ulimit settings and other tweaks here and there.

We’ve uncovered a slew of things to work on. The first obvious place was to have the static content served by Apache rather than the mongrels. This has been on the todo-list for a long time, but has always fallen to the wayside. We knocked that off and looked towards the other items that were slowing us down. That’s when I uncovered some TCPSocket weirdness within memcache-client 1.7.4 that comes with activesupport 2.3.5.

During some tests we noticed a severe lag which we narrowed down to the fact that we were pointing to a list of memcached servers that contained one that didn’t exist. Turns out that when the memcached hostname resolves, but is not pingable, memcache-client 1.7.4 waits 3 seconds before responding with an error message (in addition it doesn’t mark the server as “down”, which I think is also a bug). This 3 second delay happens on RHEL 4, and in some brief tests on Ubuntu 10.9, it was even worse, taking over 30 seconds to respond. My guess is that there is some OS level setting that affects this, but I have yet to locate it. The fun part however, is that this problem does not exist in our old environment where we run acivesupport 2.1.2 which uses memcache-client 1.5.0.

Turns out, in 1.5.0, the memcache-client uses a 250ms timeout when calling TCPSocket.new. Something that was lost on the way to 1.7.4. Some initial tests of simply adding this CONNECT_TIMEOUT back in have been promising. It’s currently not throwing the right exception or marking the server as “down”, but once I do that, I will see about posting the source somewhere.

One of the things I really like about my new job is pair programming. On Tuesday of this week, there was a bit of a panic as we tried to figure out how we were going to get work done with everyone in different locations due to the impending snow storm on Wednesday. To me, it was almost comical. I mean, I had been working from home for the last 6 or so years. I never had someone looking over my shoulder as I wrote code. One day wasn’t going to be a big deal… was it? Given the way this team has worked for the last few years, there really was no plan for how to operate when we aren’t all in the same room. There was talk of iChat and sharing screens and some other not-so-simple-to-setup environment stuff, but in the end we coordinated what we were working on via chat and agreed we’d pair up on Thursday to review changes/updates we made or wanted to make. Even though I ended up doing the same on Thursday (no plow on my street all day, I couldn’t leave the house!), I was able to be productive and reviewed my work today with some team members. It was weird working from home again. It was definitely different - I had to use my home laptop and my office space was a bit of a wreck. It took a little while to get totally setup with the VPN etc, but I ended up cranking out some good work. My final conclusion is that I really like pairing, and working in an open space. Maybe I have great teammates, maybe I’m still in the honeymoon phase of pair programming, who knows, but for now, I’m loving it.