Author Topic: yet another load game crash  (Read 2196 times)

thd

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
yet another load game crash
« on: July 29, 2015, 05:09:56 PM »
I have seen multiple topics with regards to the game crashing on load, but have not seen a single one resolved.

Every single game I have played so far, the same thing happens.  At some point, the game just crash when loading a save.

  • No mods installed
  • No custom ship designs of any kind
  • Game version 1.0.2 (steam)
  • Integrity verified

The crash does not happen with just one save file (this is not about corrupt data).  For example, I would have a save file and ~30min later the autosave.  Both crash the game.  The error is always the same: "Script Exception: Too large array size".  In fact, there are three save games over the period of an hour and they all crash the game.  A much earlier save will load successfully.

At present I am unable to play the game due to this issue and request help please.

log.txt follows (errors.log.txt is a subset of this information):
Code: [Select]
29 Jul 2015 22:56:43
Initializing sound
Initializing window system
Loading engine settings
Initializing OpenGL Engine
Registering mods
Starting 13 threads on 12 processors.
Loading mod(s)
OpenGL vendor 'NVIDIA Corporation', renderer 'GeForce GTX 970/PCIe/SSE2'
       version '4.5.0 NVIDIA 352.30'
Loaded in 2.7 seconds
Shadow scripts: 97ms load, 1179ms compile
Server scripts: 121ms load, 2435ms compile
Client scripts: 118ms load, 3304ms compile
Loaded in 3.4 seconds
Script Exception: Too large array size
 <Unknown>
  (null)::QueuedImport@[]@ factstub(uint) | Line 0 | Col 0
  components.Resources::void ObjectResources::load(SaveFile&inout) | Line 141 | Col 4
  (null)::Message& SaveFile::opShr(Savable@) | Line 0 | Col 4
  objects.Orbital::void OrbitalScript::load(Orbital&inout, SaveFile&inout) | Line 123 | Col 3

See attached for one of the (many) save files that causes the game to crash.

Neikius

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: yet another load game crash
« Reply #1 on: September 17, 2015, 03:13:18 AM »
Actually, same error here (or very similar).

Script Exception: Too large array size
 <Unknown>
  (null)::QueuedImport@[]@ factstub(uint) | Line 0 | Col 0
  components.Resources::void ObjectResources::load(SaveFile&inout) | Line 141 | Col 4
  (null)::Message& SaveFile::opShr(Savable@) | Line 0 | Col 4
  objects.Orbital::void OrbitalScript::load(Orbital&inout, SaveFile&inout) | Line 123 | Col 3

StarRuler2.bin: ../../source/as_atomic.cpp:68: asDWORD asCAtomic::atomicInc(): Assertion `value < 1000000' failed.

Don't have any mods and no designs unless they propagate.

dalolorn

  • Sentient
  • **
  • Posts: 199
  • Karma: +7/-0
  • ABEM Developer
    • View Profile
Re: yet another load game crash
« Reply #2 on: September 18, 2015, 03:46:35 AM »
Orbitals trying to load resources in 1.0.2? What nonsense is this?

This should not be happening. As far as I remember, OrbitalScript::load() doesn't try to load a Resources component unless the save is from a version where Orbitals -had- Resources components. That, or they read a boolean to determine on a case-by-case basis whether or not this particular Orbital instance had a Resources component; either way, there is nothing in a proper installation of vanilla 1.0.2 that could make this happen.

The only exceptions I can think of are A: that you don't have a proper vanilla 1.0.2, that it's been corrupted by undeleted data from previous updates or clumsily done DIY modding, or B: that this is a symptom of something else entirely going wrong with the save that causes OrbitalScript to read the wrong bits at the wrong place.

The former is (sort of) easy to fix; reinstall the game from scratch, don't rely on Steam's crappy verification. The latter, on the other hand... you'd have to figure out where the corruption takes place before you could fix the underlying issue. (Of course, the former could still be causing the latter.)

Neikius

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: yet another load game crash
« Reply #3 on: September 22, 2015, 02:49:06 PM »
Got the game on gog.com, updated via ingame update... this happens ... occasionally, but not all the time. Possibly depends on races/stars spawned? I never played anything else but 1.0.2 since I've noticed the game just recently. I will try reinstalling when I have some time.

Woos

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: yet another load game crash
« Reply #4 on: September 25, 2015, 10:34:06 AM »
I can confirm this issue.
Game also bought at GOG.
Online Update after start.
After playing for some hours savegames are corrupted (all from that point on, including autosaves). Happend already for the second time, this is getting annoying as the game is basically unplayable unless one wants to leave the computer running several days.
Code: [Select]
Star Ruler 2$ ./start.sh
Running Star Ruler 2
Initializing sound
Initializing window system
Loading engine settings
Initializing OpenGL Engine
Registering mods
Starting 5 threads on 4 processors.
Loading mod(s)
OpenGL vendor 'X.Org', renderer 'Gallium 0.4 on AMD JUNIPER'
       version '3.0 Mesa 10.1.3'
Loaded in 5.0 seconds
libpng warning: iCCP: known incorrect sRGB profile
[more of the same warning removed]
Shadow scripts: 232ms load, 1810ms compile
libpng warning: iCCP: cHRM chunk does not match sRGB
Server scripts: 316ms load, 3821ms compile
Client scripts: 292ms load, 4891ms compile
Loaded in 5.2 seconds
Script Exception: Too large array size
 <Unknown>
  (null)::QueuedImport@[]@ factstub(uint) | Line 0 | Col 0
  components.Resources::void ObjectResources::load(SaveFile&inout) | Line 141 | Col 4
  (null)::Message& SaveFile::opShr(Savable@) | Line 0 | Col 4
  objects.Orbital::void OrbitalScript::load(Orbital&inout, SaveFile&inout) | Line 123 | Col 3

StarRuler2.bin: ../../source/as_atomic.cpp:68: asDWORD asCAtomic::atomicInc(): Assertion `value < 1000000' failed.
support/gog_com.shlib: line 94:  3577 Aborted                 ./"${bin_64}"
No mods, but I do not only use the standard designs (what else in a game which is about developing ship designs).
Linux Version: Ubuntu 14.04 (although I doubt it matters except for the ICCP warnings)
Code: [Select]
Star Ruler 2$ uname -a
Linux gart 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Alternative error mode: (Causing first a long wait while seemingly memory overflows)
Code: [Select]
./start.sh
Running Star Ruler 2
Initializing sound
Initializing window system
Loading engine settings
Initializing OpenGL Engine
Registering mods
Starting 5 threads on 4 processors.
Loading mod(s)
OpenGL vendor 'X.Org', renderer 'Gallium 0.4 on AMD JUNIPER'
       version '3.0 Mesa 10.1.3'
Loaded in 4.7 seconds
....
Script Exception: Too large array size
 <Unknown>
  (null)::QueuedImport@[]@ factstub(uint) | Line 0 | Col 0
  components.Resources::void ObjectResources::load(SaveFile&inout) | Line 141 | Col 4
  (null)::Message& SaveFile::opShr(Savable@) | Line 0 | Col 4
  objects.Orbital::void OrbitalScript::load(Orbital&inout, SaveFile&inout) | Line 123 | Col 3

Script Exception: Too large array size
 /opt/games/starruler2/Star Ruler 2/game/scripts/server/components/Statuses.as
  components.Statuses::void Statuses::load(SaveFile&inout) | Line 34 | Col 3
  (null)::Message& SaveFile::opShr(Savable@) | Line 0 | Col 3
  objects.Orbital::void OrbitalScript::load(Orbital&inout, SaveFile&inout) | Line 125 | Col 3

Script Exception: Null pointer access
 /opt/games/starruler2/Star Ruler 2/game/scripts/server/orders/LeaderAI.as
  orders.LeaderAI::void LeaderAI::load(SaveFile&inout) | Line 121 | Col 4
  (null)::Message& SaveFile::opShr(Savable@) | Line 0 | Col 4
  objects.Planet::void PlanetScript::load(Planet&inout, SaveFile&inout) | Line 88 | Col 3


Unexpected Exception: std::bad_alloc
Stack trace:
 ./bin/lin64/StarRuler2.bin() [0x5cbc5a]
 ./bin/lin64/StarRuler2.bin() [0x5cbcd3]
 ./bin/lin64/libstdc++.so.6(+0x5ee56) [0x7f13ea2a0e56]
 ./bin/lin64/libstdc++.so.6(+0x5eea1) [0x7f13ea2a0ea1]
 ./bin/lin64/libstdc++.so.6(+0x5f0b8) [0x7f13ea2a10b8]
 ./bin/lin64/libstdc++.so.6(+0x5f649) [0x7f13ea2a1649]
 ./bin/lin64/StarRuler2.bin() [0x5fe2b9]
 [0x7f1323a3bfd3]

Script Exception: (null)
 /opt/games/starruler2/Star Ruler 2/game/scripts/server/orders/LeaderAI.as
  orders.LeaderAI::void LeaderAI::load(SaveFile&inout) | Line 111 | Col 3
  (null)::Message& SaveFile::opShr(Savable@) | Line 0 | Col 3
  objects.Planet::void PlanetScript::load(Planet&inout, SaveFile&inout) | Line 88 | Col 3

support/gog_com.shlib: line 94:  3953 Aborted                 ./"${bin_64}"
But appearently the triggering bug is the same.

Does your QM include loading games?

If needed I can also attach one (or several) savegames.

Woos

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: yet another load game crash
« Reply #5 on: September 27, 2015, 05:57:10 AM »
Investigated a bit more.

A) Played for several hours under Windows 7. No savefile corruption (obviously does not mean it will not occur, but at least under Linux I always had them after one or two hours).

B) It actually seems something is going wrong during save, not during load. This is what I got when starting a new game. Playing for some hours and saving from time to time. Last two saves (and arbitrary many autosaves in between, I don't know how often it autosaves) were then broken.
Code: [Select]
Script Exception: Index out of bounds
 /opt/games/starruler2/Star Ruler 2/game/scripts/server/components/Resources.as
  components.Resources::void ObjectResources::save(SaveFile&inout) | Line 63 | Col 4
  (null)::Message& SaveFile::opShl(Savable@) | Line 0 | Col 4
  objects.Orbital::void OrbitalScript::save(Orbital&inout, SaveFile&inout) | Line 56 | Col 3

Script Exception: Index out of bounds
 /opt/games/starruler2/Star Ruler 2/game/scripts/server/components/Resources.as
  components.Resources::void ObjectResources::save(SaveFile&inout) | Line 63 | Col 4
  (null)::Message& SaveFile::opShl(Savable@) | Line 0 | Col 4
  objects.Asteroid::void AsteroidScript::save(Asteroid&inout, SaveFile&inout) | Line 84 | Col 3

Script Exception: Index out of bounds
 /opt/games/starruler2/Star Ruler 2/game/scripts/server/components/Resources.as
  components.Resources::void ObjectResources::save(SaveFile&inout) | Line 63 | Col 4
  (null)::Message& SaveFile::opShl(Savable@) | Line 0 | Col 4
  objects.Planet::void PlanetScript::save(Planet&inout, SaveFile&inout) | Line 43 | Col 3

Although looking at the code mentioned
Code: [Select]
cnt = resources.length;
file << cnt;
for(uint i = 0; i < cnt; ++i) {
resources[i].save(file);
if(resources[i].usable)
resources[i].type.save(resources[i], file);
}
I have no idea how one can get an index out of bounds there unless one of the called functions (including file << cnt;) has serious side effects (or .length() is broken like e.g. returning -1 as an error report; which could get interesting results when converted into an uint).

Edit: Just in case it isn't obvious: The Problem during loading then probably occurs because the file pointer is off after the load function read objects which were never written. The exceptions during save appareantly are caught so there is a file written.
« Last Edit: September 27, 2015, 06:01:59 AM by Woos »

Woos

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: yet another load game crash
« Reply #6 on: September 29, 2015, 12:30:33 PM »
The error occurs also while running hands-off.
So in principle you could have a loadable savegame now, where the error occurs within 2 minutes.
Alas, even though the options menu allows setting the number of savegames to 30 it will always only save 10 of them, but still shows 30 in the options.
So now all the remaining 10 autosaves contain the error. And I have to try again.

But my latest theory on this error is anyway that it is caused by a race-condition ("5 threads on 4 cores" => worst kind of concurrency possible). Probably some AI is changing resource allocation while the savegame functions tries to save the data.
Nicely explains how code that cannot fail actually fails.
Doesn't really explain why all saves afterwards are also broken but my guess is that the exceptions shown in the message above kill a worker thread and afterwards only parts of the database get saved, which - due to the savegame format without any section markers - also leads to a corrupt save game.

dalolorn

  • Sentient
  • **
  • Posts: 199
  • Karma: +7/-0
  • ABEM Developer
    • View Profile
Re: yet another load game crash
« Reply #7 on: September 30, 2015, 03:16:49 AM »
The AI could perhaps change resource allocation from asteroids and planets. It could not, however, influence orbitals, and at no point could it delete resources unless it destroyed a Megafarm/Hydrogenator. (I just checked, though, and my previous post about Orbitals no longer having Resources by default was incorrect; they still have them, they just don't do anything with them.)

You're definitely right about resources.length() returning -1, though. When it wrapped around to UINT_MAX, it'd make the game try to initialize a huge array when loading the save, which fits the errors from the previous posts.

What I can't understand right now is what could possibly be convincing arrays that their length is -1 (this is an AngelScript function, so it's unlikely that the devs could have made a mistake there), and why it'd be happening on such a small subset of the game's population. It's obviously a localized issue, or there'd probably be a lot more people reporting broken savegames, but... why you?
« Last Edit: September 30, 2015, 03:42:04 AM by dalolorn »

GGLucas

  • Dr. Evil
  • BMS Staff
  • Delusional
  • *
  • Posts: 1877
  • Karma: +300/-6
    • View Profile
Re: yet another load game crash
« Reply #8 on: September 30, 2015, 06:22:55 AM »
Thanks for shedding some more light on this error, Woos. We've been aware of a (we thought very rare?) phantom corruption-on-save around there for a while, because we've gotten savegames that suddenly devolved into gibberish data, which is only caused by errors on save, but we hadn't seen the exact errors. If you do manage to create a working save that causes this error, that's pretty much a guarantee we can fix it, but until then:

I'm just as confused as you are. Probably even more, because I know for a fact that nothing runs in other threads while saving, and even if it somehow did and managed to change resources at the exact required moment, it would cause a single save operation to fail, it couldn't possibly affect all future ones. The fact that it starts happening over time is also even more puzzling; we had previously assumed that this was caused by faulty bytecode emitted by our angelscript JIT compiler, causing the count to be read/used wrong. But since the compiled bytecode doesn't change after game start, and the game saves fine up to a point, either there's something exceptionally strange going on, or it's not a problem with the script compilation.

We'll have to spend some time mashing our heads into what exactly happens here. You may be right that the array itself actually somehow receives a corrupted length :/

Woos

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: yet another load game crash
« Reply #9 on: October 04, 2015, 11:22:25 AM »
OK, I have quite some problems reproducing it in Hands-Off mode. Either no save game corruption or already the first one is corrupted.
Out of frustration I changed the code a bit to see whether I could prove my theory of external array manipulation that way. At least I got some surprise:
New loop looks like this
Code: [Select]
cnt = resources.length;
file << cnt;
for(uint i = 0; i < cnt; ++i) {
if(cnt != resources.length || i>=cnt)
print("cnt="+cnt+", lenght="+resources.length+", i="+i);
resources[i].save(file);
if(resources[i].usable)
resources[i].type.save(resources[i], file);
}
Theory: If some other tasks manipulates the array while its length is stored in cnt, the loop could fail and I want a printout. The second part of the condition (i>=cnt) I added after I got save game exceptions but no output.
With that I do get output. Lot of it:
Code: [Select]
cnt=0, lenght=0, i=0
cnt=0, lenght=0, i=0
....
Well, it might be noone else is surprised by this as they knew the used script language translates a for(; ; ) into a do {} while (in contrast to all other languages I know; although I applaud the aim for efficiency).
But I wonder why there are not more corrupted save games. What the heck is the content of resources[0] which is apparently saved when the array is supposed to have no entry?

Edit2:
I withdraw my previous guess on how the error gets produced. Reason: The above output only occurs when a corrupt save game is produced (I have about a 50% chance by using 5 players (i.e. one human & 4 AIs), each with a different drive technology and a spiral galaxies with actually 5 galaxies with about 25 planets each).
If the save is correct, no output.
Also I added a
Code: [Select]
for(uint i=0; i<0; ++i)
{
    print("i="+i);
}
into the file and that never produces any output (neither with corrupt nor correct save games).

I start to believe your idea of a problem of the compiler/interpreter.


Edit3:
Just in case it is needed, a more complete dump of the output including the Exceptions (starting at the very beginning):
Code: [Select]
cnt=0, lenght=0, i=0
Script Exception: Index out of bounds
 /opt/games/starruler2/Star Ruler 2/game/scripts/server/components/Resources.as
  components.Resources::void ObjectResources::save(SaveFile&inout) | Line 66 | Col 4
  (null)::Message& SaveFile::opShl(Savable@) | Line 0 | Col 4
  objects.Orbital::void OrbitalScript::save(Orbital&inout, SaveFile&inout) | Line 56 | Col 3

cnt=0, lenght=0, i=0
Script Exception: Index out of bounds
 /opt/games/starruler2/Star Ruler 2/game/scripts/server/components/Resources.as
  components.Resources::void ObjectResources::save(SaveFile&inout) | Line 66 | Col 4
  (null)::Message& SaveFile::opShl(Savable@) | Line 0 | Col 4
  objects.Asteroid::void AsteroidScript::save(Asteroid&inout, SaveFile&inout) | Line 84 | Col 3

cnt=0, lenght=0, i=0
cnt=0, lenght=0, i=0
cnt=0, lenght=0, i=0
cnt=0, lenght=0, i=0
cnt=0, lenght=0, i=0
Script Exception: Index out of bounds
 /opt/games/starruler2/Star Ruler 2/game/scripts/server/components/Resources.as
  components.Resources::void ObjectResources::save(SaveFile&inout) | Line 66 | Col 4
  (null)::Message& SaveFile::opShl(Savable@) | Line 0 | Col 4
  objects.Planet::void PlanetScript::save(Planet&inout, SaveFile&inout) | Line 43 | Col 3

cnt=0, lenght=0, i=0
cnt=0, lenght=0, i=0
cnt=0, lenght=0, i=0
cnt=0, lenght=0, i=0
cnt=0, lenght=0, i=0
....
The exceptions never repeat.

That's probably everything I can do. Just in case I'll run a RAM test and then I'll play the game under Windows.
« Last Edit: October 04, 2015, 12:00:48 PM by Woos »

GGLucas

  • Dr. Evil
  • BMS Staff
  • Delusional
  • *
  • Posts: 1877
  • Karma: +300/-6
    • View Profile
Re: yet another load game crash
« Reply #10 on: October 04, 2015, 12:36:20 PM »
Alright. Well that settles it then, the JIT is being an idiot somehow. Thanks for the detailed analysis, I'll definitely be spending some time looking at the assembly output for that function. Though why it would suddenly start happening is a mystery.

Neikius

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: yet another load game crash
« Reply #11 on: October 15, 2015, 04:52:36 PM »
Nice to see this is being looked at :) Do you need any assistance from me?