New release that should fix the unistd.h problem.

I just posted a new version of the library. The changes aren't very interesting. If you're a programmer, see the changelog in the zip archive for a list. If you're not a programmer, you can switch to the new version of dbpf-recompress.exe, but you probably won't notice any difference.


Oh, I know it's a spoof. But what it looks like is a domain with nothing on it but the spoof pages. You'd never guess that there's a real forum with real Sims 2 content on the same site.


Yes, that's before compression (or after decompression).

I originally posted this reply in the "New DBPF library" thread, but I'm copying it here since the original message was in both threads.


SimPE uses System.IO.BinaryReader.ReadString() and its BinaryWriter counterpart for most string serialization. These do not use a single byte for the length -- they use a weird base-128 encoding documented here: http://msdn2.microsoft.com/en-us/library/system.io.binarywriter.write7bitencodedint.aspx. In this encoding a length of 0x95 is serialized as 0x95 0x01. I'm almost certain that this is your problem, not anything to do with compression.

I don't know whether this is a bug in SimPE or whether it actually is the encoding used by The Sims. If the former, you may have to use an ugly workaround anyway.

Hi, following up on a bunch of stuff:

The problem with compressed duplicate entries is that there are two different entry lists in the file. One lists the type, group, instance, file offset, and (compressed) size for all entries, and the other lists the type, group, instance, and uncompressed size for all compressed entries. In order to figure out what's compressed you have to merge the two lists by type/group/instance, and there's no way to do that when there are duplicate entries. It's a stupid design, but it's not SimPE's fault.


SimPE uses System.IO.BinaryReader.ReadString() and its BinaryWriter counterpart for most string serialization. These do not use a single byte for the length -- they use a weird base-128 encoding documented here: http://msdn2.microsoft.com/en-us/library/system.io.binarywriter.write7bitencodedint.aspx. In this encoding a length of 0x95 is serialized as 0x95 0x01. I'm almost certain that this is your problem, not anything to do with compression.

I don't know whether this is a bug in SimPE or whether it actually is the encoding used by The Sims. If the former, you may have to use an ugly workaround anyway.


I'm not really a gamer, I'm just interested in game design. When I do play games it's usually with as many cheats and walkthroughs as I can get my hands on, which for some reason strikes many people as somehow immoral.

I think I found you through a thread in ModTheSims2. How are people supposed to find you? Or aren't they? It isn't exactly obvious how to get here from the top page, unless paying the $40 actually works.

People who are experiencing crashes: if you have the time, please try to narrow down the problem. If you can find a single file which causes a crash when it's recompressed, and post it here, I can probably fix the bug.

The command-line tool verifies that all of the recompressed files match the originals, and backs out the changes if they don't, so it should be safe, but I may have overlooked something.

Um, I guess I should have introduced myself. Hi, I'm Ben. I've never posted on a Sims forum before. I'm a PhD student in the theory of programming languages. One of my hobbies is writing decompilers for obscure virtual machines, like these: http://www.darkweb.com/~benrg/if-decompilers/. I thought that a SimAntics decompiler and recompiler might be an interesting project, and as a first step I needed a DBPF read/write library, so I looked around for one, and discovered that it was a most-wanted item, so I wrote it. Of course it took four times as long as I expected (Hofstadter's law), and the decompiler will probably never see the light of day, but in any case the library is finished (modulo bugs), so I'm happy.

As everyone else has figured out already, Jack Sparrow's hair has index entries sharing the same type/group/instance ID, and because compression status is indexed in the file by type/group/instance ID, it's impossible to have duplicate entries unless none of them are compressed. How should the library handle this? It could:

• Refuse to even open files with duplicate index entries, because they're broken. Probably a bad idea, since at least one exists.
• Open them but refuse to write them. Clients like dbpf-recompress would have to check for duplicates and... do what? Discard all but one? Compare them to make sure they're equal?
• Allow opening and writing, always writing the duplicate entries uncompressed, unless the write_disposition makes that impossible, in which case fail. This is maximally flexible but seems pointlessly complicated for a rare and useless case, and makes pointlessly larger files. It wouldn't be too hard to implement, though.

-- Ben

I think this belongs in the Bowels of Trogdor, but I don't seem to have posting permission there.

This is a DBPF (*.package) library written in C++ with a C interface, GPL licensed. It has full write support with incremental and in-place updates and compression. The compression code is based on zlib's deflate code with lazy matching, and almost always compresses better than SimPE or the game itself (sometimes a lot better). I also included a sample app / useful utility called dbpf-recompress which uses the library to recompress package files. It shaves about 100 megabytes off the size of a base Sims 2 installation, and everything seems to still work.

This is a very preliminary release and some things are completely untested. Please test dbpf-recompress on every package you can lay your hands on (after backing them up!). And please use the library to build useful tools, otherwise it will all have been in vain.

Edit: New version 20070530. See the changelog in the zip archive.

Edit: New version 20070601.