UPDATED WED -- Tech Update Monday July 13

As you are aware we've been experiencing slowness for quite some time, and sporadic "white pages" for several weeks. We are as frustrated as you, and have had several resources reviewing, digging, bug fixing, and postulating for a while now.

The white pages are being caused by database queries backing up in the system, and is reflected in a graph we monitor known as "slow queries". We monitor this in real time on each of our database servers.

Last week we rewrote or added several caching layers to the site. These primarily affected pages viewed by non-signed-in users, in order to free up capacity for signed-in users. The idea being that a thread, for example, can be re-cached every minute, thereby alleviating server load, for non signed in users, while signed in users who may wish to reply will see a live thread. This is one example of the caching we have pursued.

Over the past 2 months we have reduced the slow queries peaks week after week, but have failed to solve the consumer experience problem that manifests itself as a white page. The frequency has gone down though.

Today, Monday, we added another DB server to our cluster. It will likely go live on Wednesday. This will allow us to distribute the db queries further, reducing the chance that any one DB box will be bogged down by requests.

Other changes we have done or have ordered include replacing some network switches and interface cards, and turning off features of BigSoccer which tend to be server heavy (who is browsing, forum jump, user search).

More in a few days.

 

Re: Tech Update Monday July 13

I'm logged out and can see the grey skins. I'll check with tech.

UPDATE: The new server was successfully installed last night. We plan on bringing it online today.

 

Re: Tech Update Monday July 13

The new db server is now live. We are testing it _live_ by pulling traffic on and off the other servers. So you may see slow fast slow fast over the next little while.

 

Re: Tech Update Monday July 13

UPDATE

Reading: All logged in users are now on their own dedicated database server. All not-logged in users are on their own as well. The newer faster better-er one is for logged in. Though there are many many more non-logged in users.

Of course, when you post or pm, etc, you hit the master db. But you have to be logged in to do so.

Load levels are at their lowest in ages. We continue to tweak and monitor, and are now evaluating if we should bring another web server online to join their cousins.

Please post major complaints or speed improvements, but don't clutter this thread.

 

Re: UPDATED TUESDAY -- Tech Update Monday July 13

Yeah - We're running a db sync between the various db servers. half way done.

 

All upgrades are complete as of 3am. New servers online. New caching in place. New scripts with better queries loaded.

All tests point to problems solved and significant speed improvements. Today at noon when you are all online we will know.

 

Here is a picture to show you what happened when we went live with improvements last night.

 

let's let it work for another day first