Since the new year I’ve been working on the Fantasy NASCAR site. This has been a fun project. I spent a few weeks working on the “live” draft interface for one of the games. This included that back-end side and modifying the flash to work with the back-end. There is nothing harder than moving between XML, XSLT, Perl and then Flash.
It also required extensive upgrades to the Rhino network. New firewall, gigabit ethernet interfaces for the main routers, 30 web servers, and setup a full gigabit path to and from the Foundry ServerIron switch, and an increase of 8GB to the Oracle database server. The whole system ran flawless for 2 weeks, handling 750,000 pages views every day. Then came Daytona (the Super Bowl of NASCAR), and a minor configuration setting on the servers caused the server to over commit memory causing a race condition on all of the servers and deadlocking them. I wasted valuable time looking for network issues instead at the Apache configuration. For those Apache guys out there, I had MaxClients set to 200, which always worked with the amount of memory on the servers. For this site we moved to using mod_perl, which changed the Apache footprint from 1 MB to 60 MB, once we changed the MaxClients was set to a number that would not consume all RAM and swap the servers were up and answering all of the requests.
Stefanie wants me to tell all of you that I have been working every weekend since December to get this project completed.
For a reference point, most web site see less than 10,000 page views a month, and site like the Phoenix Zoo, during the busy months gets only 45,000 page views in a month. The Fantasy NASCAR site did over a 1,000,000 page views in one day!
Tuesday, February 19, 2008