More Awesome Than You!
Welcome, Guest. Please login or register.
2024 March 29, 06:25:29

Login with username, password and session length
Search:     Advanced search
540270 Posts in 18066 Topics by 6511 Members
Latest Member: zheng
* Home Help Search Login Register
+  More Awesome Than You!
|-+  The Bowels of Trogdor
| |-+  The Small Intestines of Trogdor
| | |-+  New DBPF library
0 Members and 1 Chinese Bot are viewing this topic. « previous next »
Pages: [1] 2 THANKS THIS IS GREAT Print
Author Topic: New DBPF library  (Read 45040 times)
benrg
Asinine Airhead

Posts: 7


View Profile
New DBPF library
« on: 2007 May 03, 21:11:52 »
THANKS THIS IS GREAT

I think this belongs in the Bowels of Trogdor, but I don't seem to have posting permission there.

This is a DBPF (*.package) library written in C++ with a C interface, GPL licensed. It has full write support with incremental and in-place updates and compression. The compression code is based on zlib's deflate code with lazy matching, and almost always compresses better than SimPE or the game itself (sometimes a lot better). I also included a sample app / useful utility called dbpf-recompress which uses the library to recompress package files. It shaves about 100 megabytes off the size of a base Sims 2 installation, and everything seems to still work.

This is a very preliminary release and some things are completely untested. Please test dbpf-recompress on every package you can lay your hands on (after backing them up!). And please use the library to build useful tools, otherwise it will all have been in vain.

Edit: New version 20070530. See the changelog in the zip archive.

Edit: New version 20070601.

* dbpf.zip (25.71 KB - downloaded 1492 times.)
* dbpf-20070530.zip (27.5 KB - downloaded 1358 times.)
* dbpf-20070601.zip (27.66 KB - downloaded 1664 times.)
« Last Edit: 2007 June 01, 15:35:47 by benrg » Logged
Theo
Lipless Loser
***
Posts: 602



View Profile
Re: New DBPF library
« Reply #1 on: 2007 May 03, 22:57:23 »
THANKS THIS IS GREAT

I'm getting the output: "*** verify failed: reopen of new file failed: bad DBPF file (duplicate entry in compressed directory)" when I run the program with this package: http://theos.chewbakkas.net/maty/Jack-Sparrow-Hair-all-colors.7z

None of the resources of that package is previously compressed.

I think this belongs in the Bowels of Trogdor, but I don't seem to have posting permission there.

It seems you now do, welcome aboard Smiley

Logged
J. M. Pescado
Fat Obstreperous Jerk
El Presidente
*****
Posts: 26281



View Profile
Re: New DBPF library
« Reply #2 on: 2007 May 03, 23:02:45 »
THANKS THIS IS GREAT

Cool. This is definitely a shiny thing. And welcome to the ranks of Peasants At Least As Tall As The Beefy Arm, whoever you are.

I'm getting the output: "*** verify failed: reopen of new file failed: bad DBPF file (duplicate entry in compressed directory)" when I run the program with this package: http://theos.chewbakkas.net/maty/Jack-Sparrow-Hair-all-colors.7z
SimPE does not like seeing files where something with the exact same instance and group values is duplicated, and then both are compressed. It makes SimPE sad.
Logged

Grant me the serenity to accept the things I cannot change, the courage to change the things I cannot accept, and the wisdom to hide the bodies of those I had to kill because they pissed me off.
twojeffs
Stupid Schlemiel
****
Posts: 1690



View Profile WWW
Re: New DBPF library
« Reply #3 on: 2007 May 03, 23:55:32 »
THANKS THIS IS GREAT

Heh. It makes Simpe barf up huge scary hairballs is what it does. This has been needed for a very long time. I'll have to give it a run.
Logged
benrg
Asinine Airhead

Posts: 7


View Profile
Re: New DBPF library
« Reply #4 on: 2007 May 04, 02:25:18 »
THANKS THIS IS GREAT

And welcome to the ranks of Peasants At Least As Tall As The Beefy Arm, whoever you are.

Um, I guess I should have introduced myself. Hi, I'm Ben. I've never posted on a Sims forum before. I'm a PhD student in the theory of programming languages. One of my hobbies is writing decompilers for obscure virtual machines, like these: http://www.darkweb.com/~benrg/if-decompilers/. I thought that a SimAntics decompiler and recompiler might be an interesting project, and as a first step I needed a DBPF read/write library, so I looked around for one, and discovered that it was a most-wanted item, so I wrote it. Of course it took four times as long as I expected (Hofstadter's law), and the decompiler will probably never see the light of day, but in any case the library is finished (modulo bugs), so I'm happy.

As everyone else has figured out already, Jack Sparrow's hair has index entries sharing the same type/group/instance ID, and because compression status is indexed in the file by type/group/instance ID, it's impossible to have duplicate entries unless none of them are compressed. How should the library handle this? It could:

  • Refuse to even open files with duplicate index entries, because they're broken. Probably a bad idea, since at least one exists.
  • Open them but refuse to write them. Clients like dbpf-recompress would have to check for duplicates and... do what? Discard all but one? Compare them to make sure they're equal?
  • Allow opening and writing, always writing the duplicate entries uncompressed, unless the write_disposition makes that impossible, in which case fail. This is maximally flexible but seems pointlessly complicated for a rare and useless case, and makes pointlessly larger files. It wouldn't be too hard to implement, though.

-- Ben
Logged
J. M. Pescado
Fat Obstreperous Jerk
El Presidente
*****
Posts: 26281



View Profile
Re: New DBPF library
« Reply #5 on: 2007 May 04, 02:39:07 »
THANKS THIS IS GREAT

I vote for D: Garbage In, Garbage Out.

And out of curiosity, do you actually play the game? How did you find us here?
Logged

Grant me the serenity to accept the things I cannot change, the courage to change the things I cannot accept, and the wisdom to hide the bodies of those I had to kill because they pissed me off.
dizzy
Souped!
*
Posts: 1572


unplugged


View Profile
Re: New DBPF library
« Reply #6 on: 2007 May 04, 07:00:32 »
THANKS THIS IS GREAT

Hi Ben. Welcome to Hell. Feel free to reuse any of my code as you wish (and I'm sure Shy won't mind if you use his). Wink

I vote for auto-removal of dupes. That seems the best policy.

I don't think Z code is obscure. I've done a bit of Inform programming, actually. Nothing public, though.
Logged

Delphy
Asinine Airhead

Posts: 20


View Profile
Re: New DBPF library
« Reply #7 on: 2007 May 09, 11:04:50 »
THANKS THIS IS GREAT

Actually, if the program is done right, the index entries wont matter - what counts is the offset to the file, not the TGI information.  A resource can have exactly the same Group and Instance but be a discrete chunk within the .package file and work perfectly fine.  Another point to be aware of is to not rely on the DIR index for the compression flag - I use a combination of this and checking the first four bytes of the chunk to make sure it's compressed.

The DBPFClass.dll I made with my Download Organiser (in C#) handles that file with no errors whatsoever, and even detects the type correctly.

What I would do is write either only the first or the last file with the same TGI (although don't forget to check high-instances too) just so you actually have part of the data.
Logged
J. M. Pescado
Fat Obstreperous Jerk
El Presidente
*****
Posts: 26281



View Profile
Re: New DBPF library
« Reply #8 on: 2007 May 09, 13:12:35 »
THANKS THIS IS GREAT

So it's just a SimPE problem, then. In that case, the current behavior is correct, and SimPE Must Be Destroyed. Option D sounds better and better.
Logged

Grant me the serenity to accept the things I cannot change, the courage to change the things I cannot accept, and the wisdom to hide the bodies of those I had to kill because they pissed me off.
dizzy
Souped!
*
Posts: 1572


unplugged


View Profile
Re: New DBPF library
« Reply #9 on: 2007 May 09, 20:52:35 »
THANKS THIS IS GREAT

My thinking on the matter is if the data is identical, then it's simply redundant and a waste of space. Ergo, in a program that reduces wasted space, it should remove that resource.
Logged

J. M. Pescado
Fat Obstreperous Jerk
El Presidente
*****
Posts: 26281



View Profile
Re: New DBPF library
« Reply #10 on: 2007 May 09, 23:18:02 »
THANKS THIS IS GREAT

Yes, but there might be a perfectly good reason why someone might save two copies of a resource: Perhaps one is a backup (since I think the second one is used as it would load last).
Logged

Grant me the serenity to accept the things I cannot change, the courage to change the things I cannot accept, and the wisdom to hide the bodies of those I had to kill because they pissed me off.
dizzy
Souped!
*
Posts: 1572


unplugged


View Profile
Re: New DBPF library
« Reply #11 on: 2007 May 15, 01:39:19 »
THANKS THIS IS GREAT

I changed a couple things to get this to compile with g++. Is there an easier way to deal with these gotos across uninitialized variables?

* dbpf-dizzy2a.zip (18.27 KB - downloaded 1120 times.)
Logged

Delphy
Asinine Airhead

Posts: 20


View Profile
Re: New DBPF library
« Reply #12 on: 2007 May 29, 09:58:41 »
THANKS THIS IS GREAT

I posted this over in the compressoriser thread but figured I'd duplicate it here.

Essentially, my DBPF code for the DDO has issues with SHPE refs in files that are re-compressed using this program.  It reads normal SHPE refs and other compressed ones fine, that I can see.

On *normal* Maxis/SimPE compressed SHPE chunks, the DDO works fine.  It is only on the SHPE chunks that are run through the Compressoriser that it borks.  The reason is that there is an extra byte inserted after the length of the sgResource string but before the actual string.

For example:

Normal compressed SHPE ref, in Hex view in SimPE: 66 42 72 61 6e 64 69... etc.  This is 1 byte for the string length (0x66 - 102 bytes) and then immediately followed by the actual string (Brandi...).
For a compressorised SHPE ref, in Hex view in SimPE: 95 01 62 72 61 6e 64 69... etc.  This is 1 byte for the string length (0x95 - 149 bytes) and then a 01 and then followed by the actual string (brandi...)

A string in the SHPE ref chunk *should* be encoded with 1 byte for the length and then the actual string.  I'm guessing Simpe either ignores this extra byte, or does something else with it - I can't find anything in the SimPE code specifically mentioning it. 

I can put a catch in the DBPFClass code to check if the first byte of compressed string is 01 and ignore it, but I'd like to know why it's there in the first place.  Perhaps my understanding of the actual string storage is not complete, but the code for the DDO works with basically every other file you can throw at it *except* the compressorised ones. Smiley

Can you shed any light as to why this extra 0x01 value is getting in there?

Regards
Delphy
Logged
J. M. Pescado
Fat Obstreperous Jerk
El Presidente
*****
Posts: 26281



View Profile
Re: New DBPF library
« Reply #13 on: 2007 May 29, 10:19:55 »
THANKS THIS IS GREAT

I do not see any extra "1" byte in my SHPEs that I compressed, but I'm using the original CLI version at the very top of this thread. Is this purely an artifact of the Jfade edition, or does it exist in the original and/or the Dizzyian edition? Or am I looking in the wrong place? Does this "DDO" (What is it?) run into the same problem reading, say, Bathroom Uses You?
Logged

Grant me the serenity to accept the things I cannot change, the courage to change the things I cannot accept, and the wisdom to hide the bodies of those I had to kill because they pissed me off.
jfade
Obtuse Oaf
***
Posts: 904


Esteemed Senator Emeritus


View Profile WWW
Re: New DBPF library
« Reply #14 on: 2007 May 29, 13:16:09 »
THANKS THIS IS GREAT

I do not see any extra "1" byte in my SHPEs that I compressed, but I'm using the original CLI version at the very top of this thread. Is this purely an artifact of the Jfade edition, or does it exist in the original and/or the Dizzyian edition? Or am I looking in the wrong place? Does this "DDO" (What is it?) run into the same problem reading, say, Bathroom Uses You?
I don't think my program has anything to do with it, as I shell out to the exact same command line version of the EXE attached at the top of this thread. I didn't recompile the code or anything, just shelled out to the same EXE, so it should be exactly the same, unless he updated it since I first downloaded the library. Maybe this one only gets added in certain circumstances?
Logged


Nifty Sims hacks and programs at: DJS Sims
J. M. Pescado
Fat Obstreperous Jerk
El Presidente
*****
Posts: 26281



View Profile
Re: New DBPF library
« Reply #15 on: 2007 May 29, 13:23:05 »
THANKS THIS IS GREAT

Not sure what circumstances those are, then. I wasn't even aware the repacker did any changes, I thought it was entirely datatype-agnostic, and performed the double-check to make sure the data uncompresses to be the same. And, like I said, I don't see it. And what's a DDO?
Logged

Grant me the serenity to accept the things I cannot change, the courage to change the things I cannot accept, and the wisdom to hide the bodies of those I had to kill because they pissed me off.
Venusy
Dead Member
*
Posts: 1392


Despite the name, I am male.Do not call me "miss".


View Profile
Re: New DBPF library
« Reply #16 on: 2007 May 29, 13:48:28 »
THANKS THIS IS GREAT

And what's a DDO?
Delphy's Download Organizer, I think there was a thread about it in the Podium a while back.

EDIT: Well, it was mentioned in a thread in the Podium a while back at least. Direct MTS2 link.
« Last Edit: 2007 May 29, 13:53:38 by Venusy » Logged

"They say only the good die young. If that works both ways, I'm immortal."

ISTP - Independent, concise in speech, master of tools.
jfade
Obtuse Oaf
***
Posts: 904


Esteemed Senator Emeritus


View Profile WWW
Re: New DBPF library
« Reply #17 on: 2007 May 29, 13:59:54 »
THANKS THIS IS GREAT

Not sure what circumstances those are, then. I wasn't even aware the repacker did any changes, I thought it was entirely datatype-agnostic, and performed the double-check to make sure the data uncompresses to be the same. And, like I said, I don't see it. And what's a DDO?
Neither was I. I'd be interested in seeing the original and compressed versions of these files to see what's up. I wonder if it's more of a problem with how the decompression itself is performed.
Logged


Nifty Sims hacks and programs at: DJS Sims
dizzy
Souped!
*
Posts: 1572


unplugged


View Profile
Re: New DBPF library
« Reply #18 on: 2007 May 29, 23:55:57 »
THANKS THIS IS GREAT

Normal compressed SHPE ref, in Hex view in SimPE: 66 42 72 61 6e 64 69... etc.  This is 1 byte for the string length (0x66 - 102 bytes) and then immediately followed by the actual string (Brandi...).
For a compressorised SHPE ref, in Hex view in SimPE: 95 01 62 72 61 6e 64 69... etc.  This is 1 byte for the string length (0x95 - 149 bytes) and then a 01 and then followed by the actual string (brandi...)

A string in the SHPE ref chunk *should* be encoded with 1 byte for the length and then the actual string.  I'm guessing Simpe either ignores this extra byte, or does something else with it - I can't find anything in the SimPE code specifically mentioning it. 

...

Can you shed any light as to why this extra 0x01 value is getting in there?

I assume you're referring to the offset+9 data stream? If so, it seems to me that expecting specific control bits for particular resources at specific offsets is a critical bug in DDO. Wouldn't a more generic unpacker routine make more sense?

Could you please provide a full example of a package that exhibits this issue?
Logged

J. M. Pescado
Fat Obstreperous Jerk
El Presidente
*****
Posts: 26281



View Profile
Re: New DBPF library
« Reply #19 on: 2007 May 30, 04:19:09 »
THANKS THIS IS GREAT

I assume you're referring to the offset+9 data stream? If so, it seems to me that expecting specific control bits for particular resources at specific offsets is a critical bug in DDO. Wouldn't a more generic unpacker routine make more sense?
Would seem more like the data is not compliant with the file format, but why this would happen is unknown to me: I don't see anything like this happening in files I've packed, and Jfade says his is just a front-end that outsources to the original command line tool.

Could you please provide a full example of a package that exhibits this issue?
And perhaps a screenshot of where you see the error?
Logged

Grant me the serenity to accept the things I cannot change, the courage to change the things I cannot accept, and the wisdom to hide the bodies of those I had to kill because they pissed me off.
benrg
Asinine Airhead

Posts: 7


View Profile
Re: New DBPF library
« Reply #20 on: 2007 May 30, 16:41:58 »
THANKS THIS IS GREAT

Hi, following up on a bunch of stuff:

The problem with compressed duplicate entries is that there are two different entry lists in the file. One lists the type, group, instance, file offset, and (compressed) size for all entries, and the other lists the type, group, instance, and uncompressed size for all compressed entries. In order to figure out what's compressed you have to merge the two lists by type/group/instance, and there's no way to do that when there are duplicate entries. It's a stupid design, but it's not SimPE's fault.

Normal compressed SHPE ref, in Hex view in SimPE: 66 42 72 61 6e 64 69... etc.  This is 1 byte for the string length (0x66 - 102 bytes) and then immediately followed by the actual string (Brandi...).
For a compressorised SHPE ref, in Hex view in SimPE: 95 01 62 72 61 6e 64 69... etc.  This is 1 byte for the string length (0x95 - 149 bytes) and then a 01 and then followed by the actual string (brandi...)

SimPE uses System.IO.BinaryReader.ReadString() and its BinaryWriter counterpart for most string serialization. These do not use a single byte for the length -- they use a weird base-128 encoding documented here: http://msdn2.microsoft.com/en-us/library/system.io.binarywriter.write7bitencodedint.aspx. In this encoding a length of 0x95 is serialized as 0x95 0x01. I'm almost certain that this is your problem, not anything to do with compression.

I don't know whether this is a bug in SimPE or whether it actually is the encoding used by The Sims. If the former, you may have to use an ugly workaround anyway.

And out of curiosity, do you actually play the game? How did you find us here?

I'm not really a gamer, I'm just interested in game design. When I do play games it's usually with as many cheats and walkthroughs as I can get my hands on, which for some reason strikes many people as somehow immoral.

I think I found you through a thread in ModTheSims2. How are people supposed to find you? Or aren't they? It isn't exactly obvious how to get here from the top page, unless paying the $40 actually works.
Logged
wes_h
Knuckleheaded Knob
**
Posts: 530


Lady on Rancho Como


View Profile
Re: New DBPF library
« Reply #21 on: 2007 May 30, 19:15:47 »
THANKS THIS IS GREAT

I think I found you through a thread in ModTheSims2. How are people supposed to find you? Or aren't they? It isn't exactly obvious how to get here from the top page, unless paying the $40 actually works.

They better cut you some slack here. Mere mortals, as opposed to talented programmers such as yourself, would get laughed at for even considering paying the $40. Mr. Pescado is of a definite antipaysite and antiestablishment mindset. Once you know this, you can then appreciate the not-so-subtle lampooning of various sacred cows that goes on here.

As for the game string length, I never noticed any longer than 127 bytes, and all my code that deals with them would suffer similar problems. Is this the way they look before compression (i.e. 0x95 0x01)?

Logged
dizzy
Souped!
*
Posts: 1572


unplugged


View Profile
Re: New DBPF library
« Reply #22 on: 2007 May 30, 23:10:01 »
THANKS THIS IS GREAT

I wouldn't be surprised. Even in Sims 1, the game engine used pstrings, cstrings, and ordinary non-terminated strings. Even worse, there was never any way you could reliably determine which format you should expect. Even resource type is NOT reliable for that determination.
Logged

wes_h
Knuckleheaded Knob
**
Posts: 530


Lady on Rancho Como


View Profile
Re: New DBPF library
« Reply #23 on: 2007 May 30, 23:26:52 »
THANKS THIS IS GREAT

I never modded Sims 1. But I just thought that all the strings that were pstrings were <128 bytes.
I have been looking at the ANIM files, and all the bone names in the file are cstrings, although you have a count already on hand of how many to look for.
I will pay more attention to this. Everything I have made so far, though, parses just specific RCOL files, not the packages.
<* Wes *>
Logged
benrg
Asinine Airhead

Posts: 7


View Profile
Re: New DBPF library
« Reply #24 on: 2007 May 30, 23:59:45 »
THANKS THIS IS GREAT

I just posted a new version of the library. The changes aren't very interesting. If you're a programmer, see the changelog in the zip archive for a list. If you're not a programmer, you can switch to the new version of dbpf-recompress.exe, but you probably won't notice any difference.

Mere mortals, as opposed to talented programmers such as yourself, would get laughed at for even considering paying the $40.

Oh, I know it's a spoof. But what it looks like is a domain with nothing on it but the spoof pages. You'd never guess that there's a real forum with real Sims 2 content on the same site.

Quote
Is this the way they look before compression (i.e. 0x95 0x01)?

Yes, that's before compression (or after decompression).
Logged
Pages: [1] 2 Print 
« previous next »
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.21 | SMF © 2015, Simple Machines Valid XHTML 1.0! Valid CSS!
Page created in 0.098 seconds with 20 queries.