How to Find Encryption Code in a Target
A Heuristic Approach

14 March 2002
by mike
Published by +Tsehp
Courtesy of Fravia's page of reverse engineering
slightly edited
There is a crack, a crack in everything That's how the light gets in
( )Beginner (x)Intermediate ( )Advanced ( )Expert

How to Find Encryption Code in a Target
A Heuristic Approach
Written by mike

Here's a little piece on how to recognize crypto code when you see it. This can help whether you're trying to break password protection, write a keygen, or understand a protocol.

Tools required
Your favorite disassembler and/or debugger

Target's URL/FTP
You know better than I ;)

First, you have to know what you're looking for. Is it hashing a password? Is it encrypting a file? Is it doing a digital signature?

Read any docs on the security of the product; often they'll name specific algorithms, which will narrow your search by a lot.

Hash functions

There are a few standardized hashes that are used all the time. MD5 and SHA are two of them. A hash function takes a block of data and mixes it into some state registers. The final state of the registers is output as the hash. The initial state, though, is always the same. You can search the exe for these constants to find the hashing code.

Hashing code is also full of small nonlinear functions; lots of and's, or's, and xor's, very densely packed. Hashing code doesn't have jumps--it's pages and pages of bit operations.

A hash will nearly always have three calls associated with it: the init function, the update function, and the finalize function. Finalize pads the end of the data out to a certain size and appends the length in bits to form the last block. This prevents certain attacks.

Stream ciphers

Encrypting a file is often done with a stream cipher like RC4. RC4 has an initialization loop where it fills a 256-byte block with the values 0-255.

Also, stream ciphers are going to XOR the plaintext with the random output bytes. I found the encryption code for WordPro by dumping the exe in WDASM. I wrote a small program to search for XORs where the first and second parameters weren't the same. I only had to glance at the code surrounding about 20 of them before I found it.

Block ciphers

There are also a few popular block ciphers that get used a lot. DES used to be popular; Blowfish, IDEA, are some others, and now that AES is in place, Rijndael will become much more popular. In order to make these run fast, there are optimizations that usually occur as precomputed tables (especially Blowfish). You can search on these popular implementations.

Block ciphers generally work on 64-bit or 128-bit chunks of data in a big loop.

Public Key

Public Key crypto is only done well with bignum code. Operations nearly always include modular exponentiation. Modular exponentiation is done in a loop over the bits of the exponent, where the number is squared, conditionally multiplied, and then reduced.

Good luck, and happy cracking!


grits to all the peudonyms at fraviamb

Final Notes
If you know what the algorithm is that you're looking for, find a public implementation, compile it, and look at the code to get a feel for it. Pay close attention to standard interfaces, like the three functions associated with hashing I mentioned before. Look at similar things (i.e. if you're looking at the MD5 hash function, compare it to SHA). Essentially, look for patterns that you'll recognize when you see them in the target.

Ob Duh

I wont even bother explaining you that you should BUY this target program if you intend to use it for a longer period than the allowed one. Should you want to STEAL this software instead, you don't need to crack its protection scheme at all: you'll find it on most Warez sites, complete and already regged, farewell, don't come back.

You are deep inside Fravia's page of reverse engineering, choose your way out:

redhomepage redlinks redsearch_forms red+ORC redhow to protect redacademy database
redreality cracking redhow to search redjavascript wars
redtools redanonymity academy redcocktails redantismut CGI-scripts redmail_Fravia
redIs reverse engineering legal?