Topic: The Forge - The Hammer of Thor (Read 10359 times)

SCM_SFHQ_XC · « **Reply #300 on:** October 21, 2007, 01:13:56 pm »

Quote from: Bonk on October 21, 2007, 11:01:56 am

Ah, how many times have you changed the missionlist? Have you edited the missionslist blob, campaign file and missions folder so they all match exactly? (I can do this for you...) Oh wait, but the stock forge is doing it too you say? (is it just as I left it? Same build?) If so, then the problem is probably not with the serverkit.

Thats what makes me confused as to where this error is coming from ... the ODBC logs dont help me at all pinpoint the problem.
No changes to missionlist, campaign file, missions folder, etc since I first set it up and yea, I tripled checked to make sure it was and indeed it is.

Dfly · « **Reply #301 on:** October 21, 2007, 04:32:30 pm »

The only 2 things I noticed on the campaign was what Lear mentioned, being that the AI markers are still present even after a clean, and the other being the loaned fighters dont show properly.

KBFLordKrueg · « **Reply #302 on:** October 21, 2007, 08:17:19 pm »

Quote from: Dfly on October 21, 2007, 04:32:30 pm

The only 2 things I noticed on the campaign was what Lear mentioned, being that the AI markers are still present even after a clean, and the other being the loaned fighters dont show properly.

The fighter thing has been a bane to the Coalition for a long time. It's becoming even more of a PITA now that the better fighters are coming out.
It's beginning to become a significant disadvantage for some ships.
Obviously, the first issue is the crashes, but, IMOHPO the fighter issue needs to be addressed right after that.
Taking into account my technical ignorance of the inner workings of a server, perhaps, the 2 issues could be related...?

Take your time with these, let's get it right rather than blundering ahead fighting constant issues.

*deposits 2 cents*

Bonk · « **Reply #303 on:** October 21, 2007, 10:41:18 pm »

I see nobody is on the server right now, so I'm going to take it down for a few minutes and go over everything. (particularly the preparedmissions table and the missionlist blob). Maybe something will jump out at me.

I think some shiplist QA might be in order. But if the "stock" forge exhibits that behavior then I guess that is out...

Ohhh waaaiiiittt. Comcast. Now filtering peer to peer traffic openly.
http://slashdot.org/articles/07/10/19/1417238.shtml
Could that be a factor? A quirk of their new torrent filtering? Hmmm...

Going to take a look now to see what I can find...

Bonk · « **Reply #304 on:** October 22, 2007, 02:31:04 am »

OK, DNS hiccups from downtime are more likely an interference than Comcasts P2P filtering.... anyway...

I went over everything. I gave it that special Bonk magic touch!

I just ran a mission, no crash. So give it a whirl... I left it running on the process sentinel...

quick summary:
1 ) reconfigured MySQL - memory, cache, named pipe, skip-innodb
2 ) reconfigured ODBC - no cache, named pipe
3 ) backed up current db
3 ) rationalised mct file, scripts folder and missionlist blob (important part! - the missionlist blob was borked)
4 ) standard db clean
5 ) backed up fixed(?) db
6 ) Did a "ServerPlatform.exe -install"
7 ) reconfigured ProcessSentinel
8 ) launched!

If you have any questions about the procedure just ask me. Hopefully it might be stable now. So get on there and give 'er a go!

odb · « **Reply #305 on:** October 22, 2007, 06:16:25 am »

approx 6.45 am monday morning ran a 6 min mission came out after debrief an i lost connection to the server. k so then i went to the oci switched from my bb to a klink d5d. i ran a 1 min 45 sec mission. came out of debrief, still connected, but when i jumped to the next hex to supply immedatly after, slow movement. took 2+ min to compleate the jump. an after the movement was done i was still in origional hex, an was still connected, an was still able to move, but i was still in the same hex after the movement was done . so i loged out, an the forge wasent showin on the server listings to log in. its now just after 7 am monday morning. i tried to make sure i ran a fast mission to see if it was a time thing that disconnected me. appearently makes no difference. vanguard was loged in ran a mission just befor i did, i came out he was gone im assuming he was disconnected to. lotta chat here that probibly dont amount to nothin just tryin to help. good luck

SCM_SFHQ_XC · « **Reply #306 on:** October 22, 2007, 07:24:08 am »

OK, thanks Bonk for fixing the Mission Blob for me

.. havent had time to log into THOR since its been brought back up, but I'll see if there anything else still clogging it up.

Bonk · « **Reply #307 on:** October 22, 2007, 09:00:12 am »

Well it looks like its down now and I'm off to work.

Hey, did I read somewhere that the characters and ships were imported into this db? Maybe we've hit the point where there is an ID conflict? Try double checking the next object ID in the database description table against the largest record ID anywhere in the db to ensure the next one is larger, I did not check that.

Bonk · « **Reply #308 on:** October 22, 2007, 03:26:13 pm »

It just occurred to me that this box Thor is on is new? It is a 64 bit AMD processor right?

We are running Win2003 Server Standard Edition on a 64 bit processor? (ummm....)

I also notice that MySQL 5.1 is installed. 5.1 is a release candidate and should not be used for production boxes. We should roll it back to 5.0.45. (dont just coopy data dirs - but dump and reload after the rollback) Also, it should be a 64 bit version of MySQL (Oonce we have the right OS installed).

From the look of it it seems your troubles started when MySQL 5.1 was intalled.

Also, we may need to recompile the serverkit for 64 bit architecture. Has that been done yet?

Also has C++ been reinstalled on thor or was the software kjust restored from a backup to the new machine? Different platform will call for a new VC install and recompilation of our applications.

(though it is acceptable to run an 32 bit OS on a 64 bit processor, it will just be a bit slower, in which case we can stick with the 32 bit apps)

SCM_SFHQ_XC · « **Reply #309 on:** October 22, 2007, 06:31:58 pm »

Quote from: Bonk on October 22, 2007, 03:26:13 pm

It just occurred to me that this box Thor is on is new? It is a 64 bit AMD processor right?

We are running Win2003 Server Standard Edition on a 64 bit processor? (ummm....)

I also notice that MySQL 5.1 is installed. 5.1 is a release candidate and should not be used for production boxes. We should roll it back to 5.0.45. (dont just coopy data dirs - but dump and reload after the rollback) Also, it should be a 64 bit version of MySQL (Oonce we have the right OS installed).

From the look of it it seems your troubles started when MySQL 5.1 was intalled.

Also, we may need to recompile the serverkit for 64 bit architecture. Has that been done yet?

Also has C++ been reinstalled on thor or was the software kjust restored from a backup to the new machine? Different platform will call for a new VC install and recompilation of our applications.

(though it is acceptable to run an 32 bit OS on a 64 bit processor, it will just be a bit slower, in which case we can stick with the 32 bit apps)

So far as I can tell V C++ has not been reinstalled on THOR, and yea I would agree.. it seems that MySQL 5.1 seems to be the thing that has changed and I can see it causing this since the serverkit was built on previous versions of MySQL. Also, yep, its a 64 bit AMD processor that THOR is running on.
I dont believ that the serverkit has been recompiled on 64 bit architecture..
it probably is the main reason for all these strange occurances with the instability of HoT.. its so sensative to every change now since it wasnt built for the latest and the best(or so MySQL 5.1 devs would claim).
I have also noticed that Process Sentinel with the serverkit running for long durations without a reset also seems to cause a slow down in preformance on the server..

Bonk · « **Reply #310 on:** October 22, 2007, 07:32:25 pm »

Quote from: SrCpt_SFHQ_XC on October 22, 2007, 06:31:58 pm

. it seems that MySQL 5.1 seems to be the thing that has changed and I can see it causing this since the serverkit was built on previous versions of MySQL.

The MySQL code was added back on EAW 2.0.36 and the current stable version of MySQL back then 3.23. The Kit code has not changed in that respect in its interaction with the SQL db, as the communication is actually through MSADO, then ODBC then to the MySQL server.

The issue is not that the serverkit was built for earlier versions of MySQL but rather that MySQL 5.1 is a release candidate only intended for testing new features.

Quote

MySQL 5.1 -- Release Candidate release, Test new features early!

MySQL 5.1 Community Edition - Release Candidate Development Release
NOTE: This release candidate release, as any other pre-production release, should not be installed on production level systems or systems with critical data. It is good practice to back up your data before installing any new version of software. Although MySQL has worked very hard to ensure a high level of quality, protect your data by making a backup as you would for any other pre release software. MySQL generally recommends that you dump and reload your tables from any previous version to upgrade to 5.1.

There is no guarantee that this is the problem, but it is mighty suspicious as it seems it was installed about when the trouble started...

Would you like me to do the rollback or will you take care of it? (Make sure to dump, uninstall, move folder, reinstall and reload dbs - also ensure no conflicting libmysql.dll is present in the system32 folder - and use a similar my.ini setup ans the changes I've made to the 5.1 config)

Bonk · « **Reply #311 on:** October 23, 2007, 01:46:37 am »

I rolled thor back to MySQL 5.0.45 and the problem persists.

So much for that theory.

In the process of working on the machine and watching the taskmanager as I went, something does not seem right about the memory in that system. I gotta ask frey if he has run a full test from the bios on these sticks in this machine at least once. Though what I'm seeing may be a result of the CPU/OS combo.

I'll check on the next object ID issue (I should have done that first)

Though this seems missions related somehow...

SCM_SFHQ_XC · « **Reply #312 on:** October 23, 2007, 04:43:00 am »

Is there any new specific error being produced since the Mission blob was fixed?
If I get the time today I'll take a look at the serverkit as well.

Bonk · « **Reply #313 on:** October 23, 2007, 11:05:50 am »

leave it be for a bit, I think I may have found the problem... I rolled it back to my last build (39) and it ran two missions in a row (an improvement) I did not test a third.

But I found 113 ships in the ship table with owner IDs not present in the character table, I removed these ships. (incomplete db cleans? - you need to remove the dead AI ships too... I have a good three query procedure for it.) This might explain that ADO/ODBC error of "could not find row to update".

So now its running under the process sentinel on build 39 with the ship table cleaned up. If its stable for a day or two then lets put build 41 back in and see what happens. And a good question would be why would build 41 be more sensitive to a bad ship table than build 39?

P.S. I have a bunch of screenshots of errors and the state of the taskmanager display while running build 41 when it was crashing, I'll try and get the relevant ones posted in the dev forum

P.P.S the DVs did not seem to be shifting properly... gotta look into that.

Bonk · « **Reply #314 on:** October 23, 2007, 12:45:35 pm »

OK, I've restored MySQL 5.1 and build 41 of the serverkit since I've established those were not the problem (though there was a difference in the kit builds)... so we can continue to test new features of MySQL as suggested!

I've got a pretty good feeling that the ghost AI ships with no AI owners in the db were the problem. Tricky one to find. I'm going to try a few missions now and see what I get!

Bonk · « **Reply #315 on:** October 23, 2007, 01:00:59 pm »

Welp, maybe not, the kit crashes again with the combo of MySQL 5.1 and build 41 of the kit.

I'll see if my old build 39 will crash with this db on MySQL 5.1...

Bonk · « **Reply #316 on:** October 23, 2007, 01:31:57 pm »

I just ran three missions in a row no troubles with kit build 39 on MySQL 5.1. I'll leave it like that for now to see if its stable.

So it was not MySQL and probably not the game db (though it had troubles) but rather the kit build 41 seems to have an issue with this game db on either MySQL 5.0.45 or 5.1.

Bonk · « **Reply #317 on:** October 23, 2007, 01:51:52 pm »

rebooting the machine...

Bonk · « **Reply #318 on:** October 23, 2007, 02:15:25 pm »

Server rebooted and back up and running. Lets see if she holds, if so, we have some debugging to do on that new build.

SCM_SFHQ_XC · « **Reply #319 on:** October 23, 2007, 06:53:46 pm »

Quote from: Bonk on October 23, 2007, 02:15:25 pm

Server rebooted and back up and running. Lets see if she holds, if so, we have some debugging to do on that new build.

Seems to be holding, though Ternado just reported to me that the Ed missions crash in nebs then causes the server to crash.

SCM_SFHQ_XC · « **Reply #320 on:** October 23, 2007, 09:01:09 pm »

OK, it seems to be crashing more often again.. noticed it crash 2 times while I was on D.net

Bonk · « **Reply #321 on:** October 23, 2007, 09:22:04 pm »

Poop, well I'm just about out of ideas...... oh wait, you could try running it on the flatfile for a while and see what happens to eliminate a few more possibilities.

FPF-Tobin Dax · « **Reply #322 on:** October 23, 2007, 10:50:10 pm »

Down again as of a few minutes ago. It wasn't offering any mission, so I thought I would log off and then back on, but it's not on the selector screen anymore.

SCM_SFHQ_XC · « **Reply #323 on:** October 24, 2007, 08:10:53 am »

Quote from: Bonk on October 23, 2007, 09:22:04 pm

Poop, well I'm just about out of ideas...... oh wait, you could try running it on the flatfile for a while and see what happens to eliminate a few more possibilities.

I'll try the switch when I get back home in a few hours.

SCM_SFHQ_XC · « **Reply #324 on:** October 24, 2007, 01:14:30 pm »

Switched to flatfile for a few hours. Please do log in and play it, and post if the server crashes. I will switch it back later tonight, but we gotta know if flatfile is stable so please dont wait for tonight for it to be switched back before you log in again. We need data!

SCM_SFHQ_XC · « **Reply #325 on:** October 24, 2007, 06:10:30 pm »

Well it seems to have remained stable on Flatfile.. so that eliminates some possibilites at least..

SCM_SFHQ_XC · « **Reply #326 on:** October 24, 2007, 08:09:24 pm »

Forge is down while THOR errors are IDed, and perhaps squashed out, as well as some internal experiments with the SQL tables

Bonk · « **Reply #327 on:** October 24, 2007, 10:05:17 pm »

Quote from: SrCpt_SFHQ_XC on October 24, 2007, 06:10:30 pm

Well it seems to have remained stable on Flatfile.. so that eliminates some possibilites at least..

OK, next I recommend running the clock up on the flatfile to the same year the MySQL db is in and see if it is still stable for the flatfile. If not, then the shiplist/missions/gfs are possibly to blame. Then you have some serious shiplist QA and gf settings to take care of.

A server can be perfectly stable on a shiplist and set of gfs in one era but not another (or one game year and not the next for that matter), it is critical to test in all years that the server will run.

Even so, if it is unstable on the flatfile in the current year, then test the current year on an older build (still on the flatfile) as well to determine if it is the build that has issues. (Dave Ferrell's build 37 is a good baseline, or build 36.)

Follow?

P.S. Did you use mission matching setting other than the default? (i.e. CaptJeff's Storm Season 2 gfs? - whic everyone but me likes, I find them very unstable - nothing personal Jeff...) If so use default mission matching settings. There are better ways to achieve what people try to do with the gf settings which are poorly understood. (trial and error)

SCM_SFHQ_XC · « **Reply #328 on:** October 25, 2007, 01:05:50 pm »

Quote from: Bonk on October 24, 2007, 10:05:17 pm

Quote from: SrCpt_SFHQ_XC on October 24, 2007, 06:10:30 pm
Well it seems to have remained stable on Flatfile.. so that eliminates some possibilites at least..

OK, next I recommend running the clock up on the flatfile to the same year the MySQL db is in and see if it is still stable for the flatfile. If not, then the shiplist/missions/gfs are possibly to blame. Then you have some serious shiplist QA and gf settings to take care of.

A server can be perfectly stable on a shiplist and set of gfs in one era but not another (or one game year and not the next for that matter), it is critical to test in all years that the server will run.

Even so, if it is unstable on the flatfile in the current year, then test the current year on an older build (still on the flatfile) as well to determine if it is the build that has issues. (Dave Ferrell's build 37 is a good baseline, or build 36.)

Follow?

P.S. Did you use mission matching setting other than the default? (i.e. CaptJeff's Storm Season 2 gfs? - whic everyone but me likes, I find them very unstable - nothing personal Jeff...) If so use default mission matching settings. There are better ways to achieve what people try to do with the gf settings which are poorly understood. (trial and error)

OK were up to 41 without any problems on flatfile, and 41 flatfile is up now on the current year.
I pretty much used the same .gf settings as Fragmentation Effect only making the necessary adjustments to add the new missions, and proper IRC chat channels etc.

SCM_SFHQ_XC · « **Reply #329 on:** October 25, 2007, 01:38:25 pm »

B41 flatfile crashing.. rerolling back to B39

News:

Author Topic: The Forge - The Hammer of Thor (Read 10359 times)