Page 1 of 1

Recent downtime

Posted: Sun Sep 30, 2007 11:47 pm
by Amp Divorax
Um, could one of the mods explain what happened to cause the bandwidth to go over and how that can be prevented?

Re: Recent downtime

Posted: Mon Oct 01, 2007 12:23 am
by lgolem
Amp Divorax wrote:Um, could one of the mods explain what happened to cause the bandwidth to go over and how that can be prevented?
It just had to happen with everyone at National Pump Fest...that is it...lol.

Posted: Mon Oct 01, 2007 12:30 am
by WhiteDragon
Ho just didn't want to have to bother with the forums while at USPF.

Posted: Mon Oct 01, 2007 12:35 am
by lgolem
WhiteDragon wrote:Ho just didn't want to have to bother with the forums while at USPF.
I don't blame him...lol

Posted: Mon Oct 01, 2007 1:59 am
by Potter
its all your damn techno music

Posted: Mon Oct 01, 2007 7:15 am
by Fluffyumpkins
Too many .gifs

Posted: Mon Oct 01, 2007 7:17 am
by Ho
I received a message on Friday morning that the site had gone over the bandwidth limit of 20 GB. I was, in fact, at USPF all weekend, but that was entirely coincidental. I was not able to login either. I couldn't even view the site control panel as I would just get the bandwidth exceeded message as well. I had to wait for the month to rollover just like everyone else. In other words, I would not have been able to do anything had I been at home.

Since I was unable to login until today, I have not had a chance to review the logs yet to see if something might have caused the bandwidth to go over. I believe the forums account for the most traffic on this site, but those should be mostly text and take quite awhile to add up to 20 GB. The photos and videos would take a lot of bandwidth, but there haven't been any new photo posts in quite awhile and the latest videos are being hosted by bacon and would not count against this sites quotas.

I intend to check later today when I get a chance to see if perhaps something went wrong or we're "leaking" bandwidth somewhere I wasn't expecting. I'll report once I've figured things out or given up in the pursuit.

As for preventative measures...maybe I'll know more once I do the above research. Otherwise, there's not really much that I can do. Hosting for this site is donated by our very own AtomX. I am really not inclined to go asking him for any more than he's already so generously given.

As for me, I don't mind the cumulative and existing recurring costs I've already invested in this site over the past ~7 years, but with basically zero revenue generation, I'm not looking to sink more money into it for something like additional bandwidth or alternate hosting. We're just going to have to find ways to work within what we've got.

Posted: Mon Oct 01, 2007 4:21 pm
by Ho
After some investigation I have determined that we got boned by the Googlebot. It accounted for almost 12.5 GB of bandwidth last month! WTF?! To give you an idea of just how out-of-whack that is, the actual site traffic amounted to about 3.75 GB. So basically the Googlebot crawled more than 3x the amount of all of us actually using the site. I'm happy to have the site indexed, but not if the result is getting it shutdown for awhile. :P

I think maybe something went wrong because the Googlebot only accounted for about 75 MB worth of traffic in August. The only way I can conceive of the Googlebot sucking down so much bandwidth is if it attempted to retrieve pretty much the entirety of the photo and video content. Even then it seems a little odd since the entire site content is only a little over 1 GB worth of files.

At any rate, I've created a robots.txt file directing bots to ignore the directory that contains all the photo JPG files and most of the video content that is still hosted here. Hopefully this will reign in the Googlebot and other crawlers. I'll monitor the bandwidth usage and see if it has the desired effect.

Posted: Mon Oct 01, 2007 5:52 pm
by Riot
I still maintain it was Chad cloggin ur tubez

Posted: Wed Oct 03, 2007 1:14 pm
by Ho
Googlebot is banned.

Even with robots.txt directing bots to not crawl the photos and videos (the largest data size items on the site), Googlebot had accounted for a cumulative ~700 MB of traffic as of yesterday and about 1.5 GB as of today. It's chugging bandwidth at an alarming rate--enough that we'd probably be shutdown mid-month if I just let it go.

After examining the logs more, it looks like Googlebot is getting caught up in the forums somehow. I'm not sure what it's trying to do, but it's gotta stop! I sent a message to Google to let them know that something is seems to be awry. But in the meantime, I've edited robots.txt to tell Googlebot to just ignore the whole site. Hopefully that will stop the bandwidth bleeding until/if Google gives me a better solution.

Posted: Tue Oct 16, 2007 2:29 am
by Amp Divorax
Oh noes, teh googlebot will starve to death without precious bandwidth to rape!!!1

I'm glad to see that the issue was eventually figured out. Hopefully Google will fix this soon.