Page 1 of 2 12 LastLast
Results 1 to 15 of 21

Thread: I need to find compression type (Data files)

  1. #1

    I need to find compression type (Data files)

    I have a game that I play, and I want to be able to extract and decompress the individual files within the large data files for modding purposes.

    MrMouse from Xantax already helped me with the extraction part, so I can extract the individual files alright. The problem is that they are still compressed, and therefore unitelligable, and there is no way to make sense of them.

    I heard that companies usually don't make their own compression for their data files, but instead they use some pre-made open src compressors. If I can just identify the method being used, then I could maybe decompress the files.

  2. #2
    I have a game that I play, and I want to be able to extract and decompress the individual files within the large data files for modding purposes.
    You will need to, if you modify, recompress. If they are using a propriatary compression, this may be very difficult.

    Assuming it's propriatary, you can get the games decompression routines quite easily. Load up Intel's VTune or AMD CodeAnalyst, run the game while sampling (stop sampling soon after decompression), and voila, the interesting parts will be pointed out.

    Load up IDA and look for places calling those locations and you'll have the outer loop of their decompressor.

    Then, load up OllyDbg, and after decompression set up a patch to dump files.

    If you believe it's a "open src compressor" (say, GNU zip or compress), you will probably see a GNU style copyright notice very close to the module they linked in. Same is usally true for 3rd party compressors. It might be a good idea to look thru the executable.

  3. #3
    I know that the game reads from these data files at the character customization screen. Then when you save your charater it outputs a file with the same compression. So all the decompression and compression happens in the character creation part of the program. I saw this with Filemon.

    For the part where it creates the character file, I set a Bp on Writefile, and then I checked the ASCII of the lpBuffer address, and it showed exactly what was written to the file, I confirmed this by opening the file in Winhex. The problem was that the ASCII in the lpBuffer had already been compressed.

    To bad there isn't a function I can bp on that has a parameter like "Data to be compressed", or "Data to be uncompressed".

    Anyway, that's as close as I got.

  4. #4
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,206
    Blog Entries
    5
    Disassemble the program and check out the code near the location where the WriteFile function is called from. Trace the buffer backwards from there in the disassembly, and it is very likely that you find your compression function quite quickly. If you don't find the decompression function at the same time (they are quite likely to be placed very near each other in the asm code) do the same with the ReadFile function for the equivalent read operation, and you should be able to find it too (but you will of course have to trace the buffer forwards after the ReadFile operation instead).

  5. #5
    You can also use hardware read breakpoints on the data read to assist, if the code is long winded. Most likely, you *will* find routines that take a couple parameters like "data to be compressed" and "buffer to put compressed data to be written" (and vice versa for the read routine). Despite all the marketing hype, there's seldom much magic to a programs innards.

  6. #6

    As Above

    Of course, you may want to rename the compressed files as .ZIP and try to decompress them with Winzip before trying anything else, as most of the compressed files (hello Quake, Quake II, Quake III, Unreal Tournamet..et al) are simply packed with the zip alogrithm.

    Have Phun
    Blame Microsoft, get l337 !!

  7. #7
    Quote Originally Posted by Aimless
    Of course, you may want to rename the compressed files as .ZIP and try to decompress them with Winzip before trying anything else, as most of the compressed files (hello Quake, Quake II, Quake III, Unreal Tournamet..et al) are simply packed with the zip alogrithm.

    Have Phun
    Do I need Winzip specifically? Or can I use Winrar?

    It would be bizarre if this actually worked, but I am skeptical.

    Note: *edit* it appears some files have diff characters.
    Last edited by Aquatic; February 10th, 2004 at 06:16.

  8. #8

    As Above

    Basically, any util that uncompresses .zip files.

    Have Phun
    Blame Microsoft, get l337 !!

  9. #9

    Hmmmm

    Just what is this game, btw?

    Have Phun,
    Last edited by Aimless; July 6th, 2004 at 00:46.
    Blame Microsoft, get l337 !!

  10. #10
    How do I trace the buffer forward/backward?

    I'm using Olly.

    Also, how will I know where the decompression code starts and finnishes?

    I will maybe post some of the code.

  11. #11
    Administrator dELTA's Avatar
    Join Date
    Oct 2000
    Location
    Ring -1
    Posts
    4,206
    Blog Entries
    5
    Debuggers aren't nearly as good as disassemblers for analyzing code, use the right tool for the right task. Disassemble it in IDA, and then you can view where the value that is pushed as a buffer parameter for the read/write calls come from, and where it goes afterwards. If that same buffer is pushed as a parameter to another function, there is a high probability that this function compresses/decompresses it.

  12. #12
    I want to follow up on this cause I still haven't done it.

    Are you saying to use a chart?

    If I do find the decompression/compression code, then how can compile it?

    I was thinking of a small Radasm app that takes a compresed file and outputs it for you in a decompressed state, and can then recompress it. But the routines that I dump will be raw ASM, so I don't know if it will compile in MASM.
    Last edited by Aquatic; July 3rd, 2004 at 15:04.

  13. #13
    When I BP on WriteFile in Olly it goes to some address starting with '77'. IDA doesn't have addresses that high.

    So how will the code even be in IDA?

    I can attatch the process with Olly, but it crashes in IDAs debugger.

  14. #14
    Sorry that means I was in kernel32.dll

    Anyway, I have had some success. I managed to go back through the code all the way to the point where clicking the 'save button' no longer triggers a breakpoint on the code.

    So should I start at the line where the first breakpoint is triggered up to the call to Writefile? Somewhere in there should be the compression code.

    I did see a lot of code that looked weird, and so it is likely that is the compression routine. The compression may span over several routines, I had to go back through about 7 or so till I got to the first line that is triggered by the saving.


    I also noticed that over time the address of the buffer will randomly change, so I have to BP on Writefile again to get the new addy.


    I will post the code from the first line that breaks up to the writefile. Maybe you guys can help me identify the code that I need to dump.

  15. #15
    If its a commercial compression algo chances are there is going to be a signature stream at the start of the file, so looking at the first 50h or so bytes of the compressed file and googling for any suspect textual data could quickly lead you in the right direction.

    No need to pull out the heavy duty toolz until you've done your background work...

    Just my 2c worth

    -Shadz

Similar Threads

  1. How to find code of type:map?
    By Darkelf in forum OllyDbg Support Forums
    Replies: 14
    Last Post: December 11th, 2008, 07:27
  2. Replies: 4
    Last Post: December 30th, 2005, 10:54
  3. Looking for compression Library jcalg
    By doug in forum Tools of Our Trade (TOT) Messageboard
    Replies: 3
    Last Post: January 12th, 2005, 17:48
  4. How to find difference between two binary files.
    By mcensamuel in forum The Newbie Forum
    Replies: 12
    Last Post: August 17th, 2004, 14:25
  5. How to trace/find files from WISE self-extracting installer?
    By cah in forum Malware Analysis and Unpacking Forum
    Replies: 3
    Last Post: October 24th, 2001, 10:50

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •