Recent downtime

Post your comments, suggestions, questions, etc. about IndyDDR.com here. Also use this forum to report issues or get help with the site.

Moderator: Moderators

Locked
User avatar
Amp Divorax
Heavy
Heavy
Posts: 2922
Joined: Wed Feb 02, 2005 3:34 pm
Location: Near some arcade place that has the same initials as Drum and Bass

Recent downtime

Post by Amp Divorax » Sun Sep 30, 2007 11:47 pm

Um, could one of the mods explain what happened to cause the bandwidth to go over and how that can be prevented?
Memorable 2016 quotes:
Ho wrote:You can break arcade games and I will fix YOU!

User avatar
lgolem
Heavy
Heavy
Posts: 2888
Joined: Sun Mar 26, 2006 1:45 am
Location: Bloomington

Re: Recent downtime

Post by lgolem » Mon Oct 01, 2007 12:23 am

Amp Divorax wrote:Um, could one of the mods explain what happened to cause the bandwidth to go over and how that can be prevented?
It just had to happen with everyone at National Pump Fest...that is it...lol.

User avatar
WhiteDragon
Heavy
Heavy
Posts: 1312
Joined: Sun Feb 13, 2005 3:04 am
Location: North of Indy
Contact:

Post by WhiteDragon » Mon Oct 01, 2007 12:30 am

Ho just didn't want to have to bother with the forums while at USPF.
Original Sin wrote:Besides, this is IndyDDR...honestly. What do you expect?

User avatar
lgolem
Heavy
Heavy
Posts: 2888
Joined: Sun Mar 26, 2006 1:45 am
Location: Bloomington

Post by lgolem » Mon Oct 01, 2007 12:35 am

WhiteDragon wrote:Ho just didn't want to have to bother with the forums while at USPF.
I don't blame him...lol

User avatar
Potter
Heavy
Heavy
Posts: 4856
Joined: Wed Feb 02, 2005 6:57 pm
Location: above the abyss
Contact:

Post by Potter » Mon Oct 01, 2007 1:59 am

its all your damn techno music
Riot wrote:My hair alone is like 5mb.
Merk wrote:Badyyyyy.. wanna go fiiiish? wanna go.... fiiiiIIIIIIIIIIIIsh?? Wanna go fishin'?!?!?! Him's a Badyyyy

User avatar
Fluffyumpkins
Moderator
Moderator
Posts: 6592
Joined: Wed Feb 02, 2005 1:53 pm

Post by Fluffyumpkins » Mon Oct 01, 2007 7:15 am

Too many .gifs

User avatar
Ho
Site Admin
Site Admin
Posts: 5645
Joined: Fri Dec 31, 2004 10:26 am
Location: The Ho-House

Post by Ho » Mon Oct 01, 2007 7:17 am

I received a message on Friday morning that the site had gone over the bandwidth limit of 20 GB. I was, in fact, at USPF all weekend, but that was entirely coincidental. I was not able to login either. I couldn't even view the site control panel as I would just get the bandwidth exceeded message as well. I had to wait for the month to rollover just like everyone else. In other words, I would not have been able to do anything had I been at home.

Since I was unable to login until today, I have not had a chance to review the logs yet to see if something might have caused the bandwidth to go over. I believe the forums account for the most traffic on this site, but those should be mostly text and take quite awhile to add up to 20 GB. The photos and videos would take a lot of bandwidth, but there haven't been any new photo posts in quite awhile and the latest videos are being hosted by bacon and would not count against this sites quotas.

I intend to check later today when I get a chance to see if perhaps something went wrong or we're "leaking" bandwidth somewhere I wasn't expecting. I'll report once I've figured things out or given up in the pursuit.

As for preventative measures...maybe I'll know more once I do the above research. Otherwise, there's not really much that I can do. Hosting for this site is donated by our very own AtomX. I am really not inclined to go asking him for any more than he's already so generously given.

As for me, I don't mind the cumulative and existing recurring costs I've already invested in this site over the past ~7 years, but with basically zero revenue generation, I'm not looking to sink more money into it for something like additional bandwidth or alternate hosting. We're just going to have to find ways to work within what we've got.
Image

User avatar
Ho
Site Admin
Site Admin
Posts: 5645
Joined: Fri Dec 31, 2004 10:26 am
Location: The Ho-House

Post by Ho » Mon Oct 01, 2007 4:21 pm

After some investigation I have determined that we got boned by the Googlebot. It accounted for almost 12.5 GB of bandwidth last month! WTF?! To give you an idea of just how out-of-whack that is, the actual site traffic amounted to about 3.75 GB. So basically the Googlebot crawled more than 3x the amount of all of us actually using the site. I'm happy to have the site indexed, but not if the result is getting it shutdown for awhile. :P

I think maybe something went wrong because the Googlebot only accounted for about 75 MB worth of traffic in August. The only way I can conceive of the Googlebot sucking down so much bandwidth is if it attempted to retrieve pretty much the entirety of the photo and video content. Even then it seems a little odd since the entire site content is only a little over 1 GB worth of files.

At any rate, I've created a robots.txt file directing bots to ignore the directory that contains all the photo JPG files and most of the video content that is still hosted here. Hopefully this will reign in the Googlebot and other crawlers. I'll monitor the bandwidth usage and see if it has the desired effect.
Image

User avatar
Riot
Heavy
Heavy
Posts: 5373
Joined: Mon Feb 21, 2005 7:58 pm

Post by Riot » Mon Oct 01, 2007 5:52 pm

I still maintain it was Chad cloggin ur tubez
Image

User avatar
Ho
Site Admin
Site Admin
Posts: 5645
Joined: Fri Dec 31, 2004 10:26 am
Location: The Ho-House

Post by Ho » Wed Oct 03, 2007 1:14 pm

Googlebot is banned.

Even with robots.txt directing bots to not crawl the photos and videos (the largest data size items on the site), Googlebot had accounted for a cumulative ~700 MB of traffic as of yesterday and about 1.5 GB as of today. It's chugging bandwidth at an alarming rate--enough that we'd probably be shutdown mid-month if I just let it go.

After examining the logs more, it looks like Googlebot is getting caught up in the forums somehow. I'm not sure what it's trying to do, but it's gotta stop! I sent a message to Google to let them know that something is seems to be awry. But in the meantime, I've edited robots.txt to tell Googlebot to just ignore the whole site. Hopefully that will stop the bandwidth bleeding until/if Google gives me a better solution.
Image

User avatar
Amp Divorax
Heavy
Heavy
Posts: 2922
Joined: Wed Feb 02, 2005 3:34 pm
Location: Near some arcade place that has the same initials as Drum and Bass

Post by Amp Divorax » Tue Oct 16, 2007 2:29 am

Oh noes, teh googlebot will starve to death without precious bandwidth to rape!!!1

I'm glad to see that the issue was eventually figured out. Hopefully Google will fix this soon.
Memorable 2016 quotes:
Ho wrote:You can break arcade games and I will fix YOU!

Locked