Kremzeek attacks Seibertron.com's database server; Autobot Mainframe later restores it!

So can yesterday’s news be restored?

SillySpringer wrote:So can yesterday’s news be restored?

Unfortunately, no. It happened late at night around 2:30am Central. Our database backups happen at 4:00am so this morning's database backup was incomplete due to the crash (and what was there was junk/unusable). My alternative to backing up to the Tuesday backup was using a backup I have from about a month ago on an offsite server. That was my last resort that I thought I was going to have to use but fortunately I was able to get the Tuesday backup to work finally after some initial problems.

Seibertron wrote:

SillySpringer wrote:So can yesterday’s news be restored?

Unfortunately, no. It happened late at night around 2:30am Central. Our database backups happen at 4:00am so this morning's database backup was incomplete due to the crash (and what was there was junk/unusable). My alternative to backing up to the Tuesday backup was using a backup I have from about a month ago on an offsite server. That was my last resort that I thought I was going to have to use but fortunately I was able to get the Tuesday backup to work finally after some initial problems.

If it might help towards any restoration I have a News RSS feed in my email, so I have the main article picture and initial article paragraph/blurbs.

Kyleor wrote:

Seibertron wrote:

SillySpringer wrote:So can yesterday’s news be restored?

Unfortunately, no. It happened late at night around 2:30am Central. Our database backups happen at 4:00am so this morning's database backup was incomplete due to the crash (and what was there was junk/unusable). My alternative to backing up to the Tuesday backup was using a backup I have from about a month ago on an offsite server. That was my last resort that I thought I was going to have to use but fortunately I was able to get the Tuesday backup to work finally after some initial problems.

If it might help towards any restoration I have a News RSS feed in my email, so I have the main article picture and initial article paragraph/blurbs.

Also just saw Google may have a cache of all the articles:

http://webcache.googleusercontent.com/s ... clnk&gl=us

I guess it doesn't come as a huge surprise since a lot of us have been getting that error that's mentioned earlier in the thread and the servers have been pretty sluggish for a while now.

Emerje

i lost all of my weapons


still got my armor and energon (but not enough to replace 8 fusion cannons)

Warning: mysql_fetch_array() expects parameter 1 to be resource, boolean given in /heavymetalwar/character-upgrade.php on line 380

i did get the upgrade tho
http://www.seibertron.com/heavymetalwar ... r_id=84224
lost all 10 repair and 10 strafe

if this is gonna happen just reset the game again already
and on my grandma's 1 year passing none the less

maybe i should just blame the cubbies

steve2275 wrote:i lost all of my weapons


still got my armor and energon (but not enough to replace 8 fusion cannons)

Warning: mysql_fetch_array() expects parameter 1 to be resource, boolean given in /heavymetalwar/character-upgrade.php on line 380

i did get the upgrade tho
http://www.seibertron.com/heavymetalwar ... r_id=84224
lost all 10 repair and 10 strafe

if this is gonna happen just reset the game again already
and on my grandma's 1 year passing none the less

maybe i should just blame the cubbies

Nice to know I'm not alone in the lost weapons/upgrades thing.

Riot Riverman wrote:

steve2275 wrote:i lost all of my weapons


still got my armor and energon (but not enough to replace 8 fusion cannons)

Warning: mysql_fetch_array() expects parameter 1 to be resource, boolean given in /heavymetalwar/character-upgrade.php on line 380

i did get the upgrade tho
http://www.seibertron.com/heavymetalwar ... r_id=84224
lost all 10 repair and 10 strafe

if this is gonna happen just reset the game again already
and on my grandma's 1 year passing none the less

maybe i should just blame the cubbies

Nice to know I'm not alone in the lost weapons/upgrades thing.

Some of these tables I'm flyin' blind on whether or not they're repaired with the correct data restored. Check out the weapons table now ... is it back to how it was a couple of days ago now?

Are you still receiving the error that you posted on the character-upgrade page?

Seibertron wrote:

Riot Riverman wrote:

steve2275 wrote:i lost all of my weapons


still got my armor and energon (but not enough to replace 8 fusion cannons)

Warning: mysql_fetch_array() expects parameter 1 to be resource, boolean given in /heavymetalwar/character-upgrade.php on line 380

i did get the upgrade tho
http://www.seibertron.com/heavymetalwar ... r_id=84224
lost all 10 repair and 10 strafe

if this is gonna happen just reset the game again already
and on my grandma's 1 year passing none the less

maybe i should just blame the cubbies

Nice to know I'm not alone in the lost weapons/upgrades thing.

Some of these tables I'm flyin' blind on whether or not they're repaired with the correct data restored. Check out the weapons table now ... is it back to how it was a couple of days ago now?

Are you still receiving the error that you posted on the character-upgrade page?

Well I got my weapons back, the character profile and upgrade pages are back to normal, but anything in training (ram, repair,etc.) is still zero on my end.

Can you post a URL to where you can see that data so that I can backtrace which tables were not restored properly? Thanks.

Riot Riverman wrote:Well I got my weapons back, the character profile and upgrade pages are back to normal, but anything in training (ram, repair,etc.) is still zero on my end.

Are the tactics restored properly now?

Riot Riverman wrote:

Seibertron wrote:

Riot Riverman wrote:

steve2275 wrote:i lost all of my weapons


still got my armor and energon (but not enough to replace 8 fusion cannons)

Warning: mysql_fetch_array() expects parameter 1 to be resource, boolean given in /heavymetalwar/character-upgrade.php on line 380

i did get the upgrade tho
http://www.seibertron.com/heavymetalwar ... r_id=84224
lost all 10 repair and 10 strafe

if this is gonna happen just reset the game again already
and on my grandma's 1 year passing none the less

maybe i should just blame the cubbies

Nice to know I'm not alone in the lost weapons/upgrades thing.

Some of these tables I'm flyin' blind on whether or not they're repaired with the correct data restored. Check out the weapons table now ... is it back to how it was a couple of days ago now?

Are you still receiving the error that you posted on the character-upgrade page?

Well I got my weapons back, the character profile and upgrade pages are back to normal, but anything in training (ram, repair,etc.) is still zero on my end.

Ok never mind my training stats are back.

Seibertron wrote:

Riot Riverman wrote:Well I got my weapons back, the character profile and upgrade pages are back to normal, but anything in training (ram, repair,etc.) is still zero on my end.

Are the tactics restored properly now?

Yes, thank you.

Question of the ignorant, this was due to hardware failure of the server?

fenrir72 wrote:Question of the ignorant, this was due to hardware failure of the server?

I don't think so. The server seibertron.com is on is pretty good. This is the longest we've ever been on 1 server before, which I think is due to it being a raid server.

I'm pretty sure this is ultimately just related to the database. To be honest though, I think I'm at the end of my abilities as an amateur server administrator. I know enough to be dangerous, enough to fix problems and enough to run Seibertron.com on a dedicated server for the past 13 years, but I'm not a database administrator. There's no silver bullet to fix the MySQL settings but I'm out of ideas. The database is still has an extreme amount of CPU usage but I'm out of ideas because it seems like nothing I'm changing is solving the problem. If anyone out there is a DBA, I'd love to talk about some suggestions and/or solutions!

Kyleor wrote:Also just saw Google may have a cache of all the articles:

http://webcache.googleusercontent.com/s ... clnk&gl=us

Gotta love Google. Thanks to your suggestion, Kyleor, and to Google's cache of Seibertron.com, all of the news stories from yesterday were restored.

http://www.seibertron.com/transformers/news/

If you see that I missed something, please let me know. Sucks we lost yesterday's forums posts, but I think we'll live.

What about the legends listings? Included Shockwave and the cassettes... (and Pink Taserpeak)

Riot Riverman wrote:

Seibertron wrote:

Riot Riverman wrote:Well I got my weapons back, the character profile and upgrade pages are back to normal, but anything in training (ram, repair,etc.) is still zero on my end.

Are the tactics restored properly now?

Yes, thank you.

http://www.seibertron.com/heavymetalwar ... r_id=84224
but he's a 9 that should be an 11
go cubbies

steve2275 wrote:but he's a 9 that should be an 11

How's that?

I think the system is still glitching. I could not get on last night for a few hours and this is the first time I've able to get on since about midnight my time

Seibertron wrote:

Kyleor wrote:Also just saw Google may have a cache of all the articles:

http://webcache.googleusercontent.com/s ... clnk&gl=us

Gotta love Google. Thanks to your suggestion, Kyleor, and to Google's cache of Seibertron.com, all of the news stories from yesterday were restored.

http://www.seibertron.com/transformers/news/

If you see that I missed something, please let me know. Sucks we lost yesterday's forums posts, but I think we'll live.

All the news articles that showed up in my RSS feed are back up.

I wonder if after posting an article you could also have the Internet Archive save a copy of the page using their 'Save Page Now' function: http://archive.org/web/

Free, immediate, article backups.

Seibertron wrote:

fenrir72 wrote:Question of the ignorant, this was due to hardware failure of the server?

I don't think so. The server seibertron.com is on is pretty good. This is the longest we've ever been on 1 server before, which I think is due to it being a raid server.

I'm pretty sure this is ultimately just related to the database. To be honest though, I think I'm at the end of my abilities as an amateur server administrator. I know enough to be dangerous, enough to fix problems and enough to run Seibertron.com on a dedicated server for the past 13 years, but I'm not a database administrator. There's no silver bullet to fix the MySQL settings but I'm out of ideas. The database is still has an extreme amount of CPU usage but I'm out of ideas because it seems like nothing I'm changing is solving the problem. If anyone out there is a DBA, I'd love to talk about some suggestions and/or solutions!

I know I've mentioned my IT Background previously, but it is mostly in infrastructure admin/support, so I am limited in DB, and most of my knowledge comes from SQL, not PHP/MySQL. (I have some second hand anecdotes from that realm, but they are dated....like ten years dated.)

The only advice I could offer right now is basic DB admin. Does MySQL offer database defragmentation? In SQL, I believe the DB has to be taken offline during this process. The DBAs I know usually find an off-peak window monthly (or even weekly for one of my larger companies) to do this maintenance task. This could be of help, if you didn't already know about it. (If you do, sorry for the redundancy as I did not want to assume knowledge level.) Also, are you running the DB on a single server with RAID? A public or private cloud option might be a possibility which would offer some redundancy in clustering with fail-over. But this could transition the cost from a single capital expense (the cost of the server setup) to a subscription based cost with a monthly fee.

When I have some free time I'll see if I can reach out to some friends who have a bit more MySQL experience to ask some of these basic questions. From a hardware guy perspective, I suspect the high CPU use has to do with the size of the DB and number of records to maintain. But, that is just a suspicion.

Happy admining!

The Kremzeek dance party will go on for how long?

Agamemnon wrote:

Seibertron wrote:

fenrir72 wrote:Question of the ignorant, this was due to hardware failure of the server?

I don't think so. The server seibertron.com is on is pretty good. This is the longest we've ever been on 1 server before, which I think is due to it being a raid server.

I'm pretty sure this is ultimately just related to the database. To be honest though, I think I'm at the end of my abilities as an amateur server administrator. I know enough to be dangerous, enough to fix problems and enough to run Seibertron.com on a dedicated server for the past 13 years, but I'm not a database administrator. There's no silver bullet to fix the MySQL settings but I'm out of ideas. The database is still has an extreme amount of CPU usage but I'm out of ideas because it seems like nothing I'm changing is solving the problem. If anyone out there is a DBA, I'd love to talk about some suggestions and/or solutions!

I know I've mentioned my IT Background previously, but it is mostly in infrastructure admin/support, so I am limited in DB, and most of my knowledge comes from SQL, not PHP/MySQL. (I have some second hand anecdotes from that realm, but they are dated....like ten years dated.)

The only advice I could offer right now is basic DB admin. Does MySQL offer database defragmentation? In SQL, I believe the DB has to be taken offline during this process. The DBAs I know usually find an off-peak window monthly (or even weekly for one of my larger companies) to do this maintenance task. This could be of help, if you didn't already know about it. (If you do, sorry for the redundancy as I did not want to assume knowledge level.) Also, are you running the DB on a single server with RAID? A public or private cloud option might be a possibility which would offer some redundancy in clustering with fail-over. But this could transition the cost from a single capital expense (the cost of the server setup) to a subscription based cost with a monthly fee.

When I have some free time I'll see if I can reach out to some friends who have a bit more MySQL experience to ask some of these basic questions. From a hardware guy perspective, I suspect the high CPU use has to do with the size of the DB and number of records to maintain. But, that is just a suspicion.

Happy admining!

I've had the high CPU usage problem here at work. It was being caused by the Sophos SQL database which in turn was playing havoc with all the other SQL databases (which our main piece of software runs on) so much so the entire network was crashing.

Once I stop Sophos, it's all good.

A ton of changes were made yesterday and today to various configuration files. Thanks to my brother-in-law for coming over to help me look through everything yesterday for about 10 hours or so. Spent probably another full day of working on changes to the server today. Seems to be working much better after the past two days. Hopefully things are OK for the time being. Fingers crossed!

Also made a new error page for when the database doesn't connect properly instead of the scraplets error message. Something a little more visual for those dreaded moments.

http://www.seibertron.com/errors/kremzeek.php

Seibertron wrote:A ton of changes were made yesterday and today to various configuration files. Thanks to my brother-in-law for coming over to help me look through everything yesterday for about 10 hours or so. Spent probably another full day of working on changes to the server today. Seems to be working much better after the past two days. Hopefully things are OK for the time being. Fingers crossed!

Also made a new error page for when the database doesn't connect properly instead of the scraplets error message. Something a little more visual for those dreaded moments.

http://www.seibertron.com/errors/kremzeek.php

We appreciate the creativity!

Aside from a brief slow down about 40 minutes ago that may have been addressed by yet another round of minor changes, how has the site's speed been today for you guys?

Seibertron wrote:Aside from a brief slow down about 40 minutes ago that may have been addressed by yet another round of minor changes, how has the site's speed been today for you guys?

Speed looks good, but now my Norton toolbar suspects the site of phishing which has never happened before.

Riot Riverman wrote:

Seibertron wrote:Aside from a brief slow down about 40 minutes ago that may have been addressed by yet another round of minor changes, how has the site's speed been today for you guys?

Speed looks good, but now my Norton toolbar suspects the site of phishing which has never happened before.

That's odd. Can you add Seibertron.com to a whitelist to keep norton from doing that?

Seibertron wrote:

Riot Riverman wrote:

Seibertron wrote:Aside from a brief slow down about 40 minutes ago that may have been addressed by yet another round of minor changes, how has the site's speed been today for you guys?

Speed looks good, but now my Norton toolbar suspects the site of phishing which has never happened before.

That's odd. Can you add Seibertron.com to a whitelist to keep norton from doing that?

No list to speak of, but there is a review page at https://safeweb.norton.com/, a place for site owners: https://safeweb.norton.com/help/site_owners, and I can just click "go to page anyway" when it blocks the path.

Hey Boss, I know you're busy.

A couple things I've noticed last night and this morning.
No e-mail notifications are going through.
Site is really sluggish.
And apparently the fucking swear filters are on the fritz.

Hope you haven't pulled out all your hair.
Good luck.

fuck

fuck cunt

Cobotron wrote:Hey Boss, I know you're busy.

A couple things I've noticed last night and this morning.
No e-mail notifications are going through.
Site is really sluggish.
And apparently the **** swear filters are on the fritz.

Hope you haven't pulled out all your hair.
Good luck.

Thanks for keeping me posted about how things are going on your end.

I fixed the word censor filter (somehow they were lost along the way, but were restored from a backup I had locally).

Site speed should be all good for now. I have a larger post about that I'll make soon after keeping an eye on things. Let me know how it's working for you now.

I'll look into the email notifications not going through. I'm not sure what's causing that problem yet.

Cobotron wrote:No e-mail notifications are going through.

Fixed! I had to reinstall the mail server along with a few other things. I've already been receiving notifications, hopefully you will receive this one from my reply. Thanks again for standing by and supporting Seibertron.com, Cobotron!

Seibertron wrote:

Cobotron wrote:No e-mail notifications are going through.

Fixed! I had to reinstall the mail server along with a few other things. I've already been receiving notifications, hopefully you will receive this one from my reply. Thanks again for standing by and supporting Seibertron.com, Cobotron!

Loud and clear!
And thanks for the edit on my OG post. I'd planned on cleaning that up once you'd seen it.

Keep it up. You're doing great!
Thanks

Cobotron wrote:

Seibertron wrote:

Cobotron wrote:No e-mail notifications are going through.

Fixed! I had to reinstall the mail server along with a few other things. I've already been receiving notifications, hopefully you will receive this one from my reply. Thanks again for standing by and supporting Seibertron.com, Cobotron!

Loud and clear!
And thanks for the edit on my OG post. I'd planned on cleaning that up once you'd seen it.

Keep it up. You're doing great!
Thanks

I didn't need to clean up anything, my friend! phpBB does it all on its own.

Now that the problem with phpBB is finally out of the way, I'm able to more clearly see what code of mine might need to be adjusted. How are the galleries loading for you guys now? Here's an example ...

http://www.seibertron.com/transformers-toys/gallery/exclusives/g2-starscream/3868/1/

Seibertron wrote:Now that the problem with phpBB is finally out of the way, I'm able to more clearly see what code of mine might need to be adjusted. How are the galleries loading for you guys now? Here's an example ...

http://www.seibertron.com/transformers-toys/gallery/exclusives/g2-starscream/3868/1/

Everything is going good as of this moment. All is loading at speed and no crashes yet

Seibertron wrote:I didn't need to clean up anything, my friend! phpBB does it all on its own.

Good! Now I can stop chasing down potty mouths.

And that gallery loaded faster than the speed a light.

There still seems to be an issue with the forum sessions database table. I can't figure out what the problem is yet. Sorry I had to log all of you out. I had to clear out all of the sessions to sync up these two tables to eliminate that as an issue.

Just a heads up, kept getting an SQL error yesterday. And a post (which I reposted in the TAV again) didn't register. It had nothing controversial in it so I don't think it was a mod who deleted it, just a request for payment of an available product.

Maybe kremzeek is active again?

i fucking hate all of these goddamn server problems!!!!!!!!!!!!!!!! i want my life back!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Sigh... Here we go again. Not only lately I keep getting logged out, but I no longer receive notifications for threads anymore.

Hadn't spotted this thread before. I've already sent a message, but i'll report it here too.
When i access the Toys Discussion forum, i won't be logged in and when i log in and access it, it will log me out.
I don't have this problem with the other forums i frequent (general, non-tf). When i access those pages from outside or inside Seibertron, i stay logged in

On the forum screen, the last posts on a specific topic doesn't actually reflect the latest topic posted.

Just a feedback Ryan.

switched 5 primary forum tables back to ENGINE=MyISAM after switching to InnoDB last night. Not sure what went wrong with those 5 tables over night.

Seems like the boards (but not the rest of the site) were down all day until now.

Sabrblade wrote:Seems like the boards (but not the rest of the site) were down all day until now.

They have been down for me for the past 14 hours. the main site page was accessible, but for some reason it included the 3 new articles for today at one point then it went back in time and only showed the news up to last night, so i am just now getting to the new stories.

Making progress with things on the server. It should be stable for the next 12 to 14 hours after the latest round of changes. There's an automated script that runs around 4am that might cause a problem. If it doesn, I'll know what the last major culprit is. If it's all good at 4am, then I'll know the changes I've made during the past 48 hours to various scripts and server settings resolved one of the problems.