Serial File Load Limit

[X3N0Wolf]

Hey all, I just ran across something in the course of my project, don't know if this has already been brought up, but it seems you can only load a certain number of serialized data files into a project, 509 to be exact. After that, somethings wigs out and it starts claiming it can't find the Serial file it's looking for even if it's plainly there. Anyone else noticed this? I wrote a dead simple bit of test code to be sure:

NDB_NEWTONCREATE

FOR I = 1 TO 510
  COLLISIONDATA = NDB_NEWTONCREATETREECOLLISIONFROMSERIALIZATION("SERIALFILE.SER")
  NEXT I

CENTER TEXT SCREEN WIDTH() / 2, SCREEN HEIGHT() / 2, "COMPLETED"
WAIT KEY

and sure enough, if I reduce the loop maximum to 509, the program displays 'completed' with no gripes, but if I leave it at 510, it complains that it can't find the file. Increase it to 511 and it'll complain twice, and so on.

I've tried it with both a simple and a more complicated serial file, thinking it might be a memory issue, but it does the same with both (and I have a gig of RAM, surely it couldn't be using it all even with MAX 5 running in the background too?).

I'm using Wrapper version 1.32, files were serialized using my own quick Serializer I wrote.

It's the wrapper that's complaining as far as I can tell, as the error window is an NDP Newton Wrapper one. Can anyone else confirm this issue? And is it possible to get around it somehow <looks hopeful>? 510 is a weird number for it to freak out on, I thought it might be 512 but no... Any help would be much appreciated guys!

Thanks!

Wolf

[kjelle69]

Went through the code and I saw that this could be an error induced by limits in open files. See the code never closes the hook to the file.

I will try to fix this and put a new archive up for download. Can you try it out to see if it works Wolf?

(I will tell you when it's available for download, probably in a couple of days.)

[X3N0Wolf]

Great stuff, thanks kjelle! Yep, let me know when you have a fixed version and I'll stress test it lol if it can be broken, I'll find a way to do it!

Cheers mate!

Wolf

[kjelle69]

It is .....




*DonE*


Try it

[X3N0Wolf]

aaand the answer is...


IT WORKS!!!


kjelle you are the MAN! I cranked it up to load the test file 99999 times, just to be sure, now THAT took a while to complete, but it did it! Fantastic, thanks for the rapid fix dude, if we ever end up in the same alcohol-dispensing establishment, I owe you a hangover! ;D

Cheers mate!

Wolf

[X3N0Wolf]

Oh yeah, kjelle, I had a thought while I was pondering this problem... would it be possible to add Clone/Instance Collision Data commands to the wrapper? In the same way that you can Clone/Instance DBP objects? 'Cuz I was thinking, it's probably quicker to reference data that's already in memory, rather than re-reading it from the hard drive again, isn't it? Loading serial files is fast, but could duplicating data already in memory be even faster? Would be useful for things like my current project where I'm loading up loads of identical boulders for scenery, with a few hundred it causes a noticeable pause while it loads the serial files. OK, I know I'm power-mad, but I want INSTANT loading dammit! lol actually, regenerating the level as near-instantly as possible is a big part of my game, so, I thought I'd ask Just a thought anyway, it could help us squeeze a few more horsepower out of our code that's all

Cheers!

Wolf

[kjelle69]

Thanks for the feedback :-) It's all in a days work ;D

About the later question: You mean like a CloneTreecollision. hmm.

I will take some time to think about it, it should be possible I guess, however, this will not be done in a day :-) That I can promise.

[Kjelle69]

just a note- if several objects have the same shape, you can re-use the collision between them. for example if you make a collision from a convex hull, you can make 50 bodies all from that same collision data.

[kjelle69]

Is it the same way with the treecollisiondata?

If not, there may be a faster way to use those two newton commands in one wrapper command:

Pseudocode:

Wrapper command: NDB_CloneTreecollision (

Assign handle to memory area ?

NewtonTreeCollisionSerialize(
const NewtonCollision* treeCollision,
NewtonSerialize callback,
void* serializeHandle)

Point the serialize handle to the memory area instead of to a file. And then read in the serialization from that memory area.

NewtonCreateTreeCollisionFromSerialization(
const NewtonWorld* newtonWorld,
NewtonTreeCollisionCallback userCallback,
NewtonDeserialize deserializeFunction,
void* serializeHandle)


Does it sound like an idea worth trying? The question is how to make a handle that points to a memoryarea which acts like a file ?

[X3N0Wolf]

No, wait, Walaber's right. I'm being a moron. I could do this:

DIM OBJECT(150)

COLLISIONDATA = NDB_NEWTONCREATETREECOLLISIONFROMSERIALIZATION(SERIAL.SER)

FOR I = 1 TO 150
OBJECT(I) = NDB_NEWTONCREATEBODY(COLLISIONDATA)
NEXT I

couldn't I? The only challenge would arise because my scenery is randomly generated, but, can I have multiple collision data slots open at once? Such as:

COLLISIONDATA1 = NDB_NEWTONCREATETREECOLLISIONFROMSERIALIZATION(ROCK1.SER)
COLLISIONDATA2 = NDB_NEWTONCREATETREECOLLISIONFROMSERIALIZATION(ROCK2.SER)
COLLISIONDATA3 = NDB_NEWTONCREATETREECOLLISIONFROMSERIALIZATION(ROCK3.SER)

and so on, and then reference the appropriate COLLISIONDATA slot when scenery of that type is generated? Does that sound do-able? Thanks for the thoughts anyway, this is what a community is all about!

Wolf

[Kjelle69]

Wolf - that's exactly right. you can re-use ANY collision object, including compound collisions, treecollisions, etc.

do it just like you said and you should have no trouble.


most people don't realize that rigid bodies and their collision geometry are seperate in Newton. you can even change the collision geometry in realtime if you want. for example you can have 2 collision shapes, representing your spaceship in "normal" and "extended wings" modes. when the user turns on "extended wings", you simply call the NewtonBodySetCollision command for the other shape, and it changes instantly.

[X3N0Wolf]

Wow, now there's a concept! Cheers Walaber! Yep, I modified my code to load each serial file once at startup, and hold the collision data in an array. Then when scenery of a certain type is generated, it just grabs a copy from the appropriate slot I never thought of it before because I'm so used to using my standard block of code that releases the collision data at the end, as you say, I plain forgot that it's a two-stage process creating collision data and that the first stage can be kept and re-used

kjelle, thanks for entertaining my crazy ideas too! ;D

Cheers guys!

Wolf

[IceNine5]

On a related note, the multilimb serializer I've been building has hit a wall - it seems there's some arbitrary limit on treecollision filesizes or geometry, or some internal quirk I'm not aware of ATM.

It runs perfectly up until about...499709 faces as close as I can tell ( dead reckoned it to 499000 then single stepped it the last bit ) after which the app simply hangs without an error message. I can attach or mail the code if that'll help locate the problem ( the progress bars are busted as it sits right now ). I walked it through without actually creating the serialization to debug the flow and it went all the way no problem, but once I added the treebuilding it choked out again.

It *could* be a RAM issue, I only have 256Mb installed, but I'll know for sure in a day or so when I have time to install the other sticks sitting on my worktable, if so I'll re-post with an update.

[IceNine5]

Nope, not a RAM issue. Couldn't sleep, so I stuffed the new RAM in and tried it. Chokes out at the same amount, and spits this error -

SERIALIZER_B caused an invalid page fault in
module NEWTON.DLL at 016f:053b54d0.
Registers:
EAX=053ea398 CS=016f EIP=053b54d0 EFLGS=00010202
EBX=00000000 SS=0177 ESP=00f4f4d0 EBP=00f4f504
ECX=00f4f4dc DS=0177 ESI=16cb5bb0 FS=131f
EDX=c1a86500 ES=0177 EDI=053ea360 GS=0000
Bytes at CS:EIP:
39 7e 20 74 09 8b 76 0c 85 f6 75 f4 eb 77 e8 1d
Stack dump:
00000000 053ea360 053ea360 00f4f54c 053c2168 00000000 053b0e28 00000001 053c2470 00000001 053b9358 00000000 00000000 00f4f520 053b93e3 00000000

[Kjelle69]

Icenine, are your serializer ready to use ? I would be very interested in testing it !