Information Hiding Methods used by FLEXlm Targets
An Explanation of the FLEXlm Seed Hiding System.
Written by Nolan Blender


This document explains how FLEXlm hides the seeds so that casual cracking techniques cannot find the seeds. An explanation of how the information can be extracted is included as well. FLEXlm has undergone a gradual evolution. Due to pressure from crackers the programmers who have written this code have improved the quality of the protection it provides. At the current stage of product evolution, it is no longer possible to simply extract the seeds from the lc_init call, even if in reality it is called from the lc_new_job call. This essay will explain the new methods used by FLEXlm to hide relevant information needed to build licenses.

Tools required

SoftICE, Hiew, DDE standard unix toolset, FLEXlm SDK 6.1g.

Target's URL/FTP

Program History

There isn't too much history yet, but it is possible to see how FLEXlm has evolved. Early versions of FLEXlm simply included the seeds and didn't encrypt them. Later versions used vendor_code5 (which was not passed into lc_init) to prevent easy discovery of the key. The most recent versions (at the time of this writing, 6.1 to 7.0c) use lc_new_job() in distributed binaries to hide the seeds.


For a full understanding of of how lm_new_job() works, I suggest that you read the essay by Dan, as it will provide you with a complete understanding of how the hiding is used in FLEXlm licensed targets. This essay provides auxillary information that will clarify exactly how the random data in the job structure is combined with the vendor code data.

The current version of FLEXlm uses special techniques to hide the encryption seeds required to generate licenses. Earlier versions simply stored the information in global data, then passed the information into lc_init(). No attempt was made to disguise the information, so of course examining the initialized global area would reveal the seeds and the vendor codes. From that point, it was a simple matter to extract vendorcode5 and xor it with data[1] and data[2], which would reveal the original seeds. Later versions hid the information using lm_new.c.

The process for storing the encrypted data involves storing one byte chunks of data in the global space, and then reassembling the data in the routine pointed to by l_n36_buf. There is a lot of code which does not get executed, however the references to the symbols is enough to keep the filler globals from being compiled out. The vendorname and the vendor keys that are returned are correct. The encryption seeds are not set to the real encryption seeds, or for that matter, the encryption seeds xored with vendor key5, but the encryption seeds xored with a random value, and then salted with values from the vendor name and the first two vendor keys. The other task that this program performs is to set l_n36_buff (note: two 'f's) to the decrypting function.

The routine takes two arguments, buf and v, where buf is a char * pointer that will have the vendor name in it after the function is called. v is a pointer to a vendorcode structure that will be filled in, but the encryption seeds will have been xored wih a random value. The decrypting routine takes in 3 values; the job structure (a shared data structure which is passed around between routines), the vendor_id, and a pointer to the vendorcode structure, which contains the randomized seeds and the correct vendor keys.

There are two modes of operation for the decrypting routine. If a null pointer for job is passed into the routine, the routine extracts the correct values for the encryption seeds and stores them in the vendorcode structure. If a valid pointer is passed in for job, time based randomized data is stored in the 12 bytes starting at v+8, then the returned seeds xored with a select set of those 12 bytes. That way the clear seeds are not available until a routine that knows how to xor the correct 4 bytes with the vendor keys is called.

A careful examination of the decryption routine in lm_new.c shows many operations, and the seeds are xored with many values. The main point of interest from a seed extraction process is the selection of the bytes from the job structure, since knowing that will enable us to get the correct seeds from the structure.

key->data[0] ^=

(((((long)sig[0] << 0)|

((long)sig[1] << 2) |

((long)sig[2] << 3) |

((long)sig[3] << 1))

^ ((long)(t->a[5]) << 0)

^ ((long)(t->a[0]) << 8)

^ x

^ ((long)(t->a[1]) << 16)

^ ((long)(t->a[4]) << 24)

^ key->keys[1]

^ key->keys[0]) & 0xffffffff) ;

sig, which is a value generated from the vendor_id, is shifted and ored with the encryption seed, as well as the randomized value x. This really doesn't matter, since this is accounted for already in the n_36_buf routine. The indices used for the t->a[%d] values are what is of interest here. The routine uniqcode in lmrand2.o was examined, and the the code which generates the decrypting program was partly reversed. It turns out that the first character of the vendor name is extracted, then that value mod 20 is used to select the bytes used to decrypt the seeds. Rather than reverse the entire routine, a program to generate vendor codes was written based on earlier reversing techniques, and an lm_new.c was generated for each of the 20 different possibilities. The values were then extracted, and a table containing these values was generated. I'm certain that the table could have been extracted from the lmrand2.o file, but it was easier to get the data from runs of lmrand2. You can use SoftICE or DDE or adb to see what is being done, the selection code happens right at the beginning of uniqcode.

Link this demonstration program with lm_new.obj, and run. This program has been tested on HPUX using the ANSI compiler, and under Windows NT using Microsoft Visual C++ 5.0.

#include <stdio.h>



* Module: extr.c v1.0.0.0


* Description: Demonstration program to show interaction

*              between security code in lm_new.c and main

*              program.


* blendern

* 08-oct-1999


* Last modified: 08-oct-1999



* Decoding table type


typedef struct decode_table_s {

        int offsets[4];

} decode_table_t;


* really vendorcode5, but renamed to avoid conflict

* if Globetrotter headers are included.


* The encryption seeds are stored in the data part of this structure.


* the VENDOR_CODES are stored in the keys part of this structure.


typedef struct my_vendorcode5 {

                            short type;    /* Type of structure */

                            unsigned long data[2]; /* 64-bit code */

                            unsigned long keys[4]; /*- [0]: Product features 


                                                   /*- [1]: platforms */

                                                   /*- [2]: platforms */

                                                   /*- [3]: Expiration date/ 


                                                   /*-      Key check */

                            short flexlm_version;

                            short flexlm_revision;

                            char flexlm_patch[2];

#define LM_MAX_BEH_VER 4

                            char behavior_ver[LM_MAX_BEH_VER + 1];

                          } MY_VENDORCODE5;


* decryption table values.


decode_table_t decode_table[] = {

        {3, 5, 4, 11}, /* index 0 */

        {9, 8, 3, 1}, /* index 1 */

        {8, 1, 2, 5}, /* index 2 */

        {2, 1, 10, 5}, /* index 3 */

        {3, 0, 1, 7}, /* index 4 */

        {1, 10, 3, 7}, /* index 5 */

        {7, 3, 5, 11}, /* index 6 */

        {0, 1, 9, 4}, /* index 7 */

        {0, 4, 1, 10}, /* index 8 */

        {11, 8, 1, 3}, /* index 9 */

        {8, 4, 2, 5}, /* index 10 */

        {6, 1, 0, 9}, /* index 11 */

        {4, 3, 8, 9}, /* index 12 */

        {0, 4, 2, 10}, /* index 13 */

        {3, 10, 8, 7}, /* index 14 */

        {1, 11, 0, 3}, /* index 15 */

        {6, 5, 1, 0}, /* index 16 */

        {0, 2, 4, 8}, /* index 17 */

        {5, 0, 1, 4}, /* index 18 */

        {10, 3, 5, 1} /* index 19 */


/* function that we get from lm_new.c */

extern int (*l_n36_buf)();

/* function pointer that will be set later. */

void (*l_n36_buff)();



* extr: demostrate calls to lm_new



void main(int argc, char *argv)


        int retcode;

        int i;

        int key_index;

        int tabindex1;

        int tabindex2;

        int tabindex3;

        int tabindex4;

        int decrypt_xor_value;

        MY_VENDORCODE5 teststruct;

        char myvendorname[1024];  /* This is the vendor_id */

        /* Job structure */

        struct s_tmp


                int i;

                char *cp;

                unsigned char a[12];

        } myjob;

        /* Initialize job */

        myjob.i = 66;

        myjob.cp = 0;


         * ensure that l_n36_buff is null, since lm_new tests

         * for this before setting it.


        l_n36_buff = 0;

        /* Get vendorname with clear keys but encrypted seeds */

        retcode = (*l_n36_buf)(myvendorname, &teststruct);

        if (retcode != 1)


                printf ("Problem with call to lm_new initializer.\n");

                printf ("retcode = %d\n", retcode);



        /* DEBUG see what was returned */

        printf ("After call to l_n36_buf routine.\n");

        printf ("myvendorname: %s\n", myvendorname);

        for (i = 0; i < 2; i++)


                printf ("data[%d] = %08x\n", i,[i]);


        for (i = 0; i < 4; i++)


                printf ("keys[%d] = %08x\n", i, teststruct.keys[i]);


        (*l_n36_buff)(&myjob, myvendorname, &teststruct);


         * extract the xoring value for the job now.


         * the index into the decoding table is the first

         * character of the vendor name mod 20.


        key_index = (int)(myvendorname[0] & 0xff) % 20;

        tabindex1 = decode_table[key_index].offsets[0];

        tabindex2 = decode_table[key_index].offsets[1];

        tabindex3 = decode_table[key_index].offsets[2];

        tabindex4 = decode_table[key_index].offsets[3];


         * use values to reverse xor which occurs if

         * valid job given.


        decrypt_xor_value = ((long)(myjob.a[tabindex1]) << 0)

                | ((long)(myjob.a[tabindex2]) << 8)

                | ((long)(myjob.a[tabindex3]) << 16)

                | ((long)(myjob.a[tabindex4]) << 24);

        printf ("Decrypt xor value: %08x\n", decrypt_xor_value);

        printf ("seed1: %08x\n",[0] ^ decrypt_xor_value);

        printf ("seed2: %08x\n",[1] ^ decrypt_xor_value);



Here is one run of the program; you will get different results for the decrypt xor value for each run, however the seeds will be the same each time.

After call to l_n36_buf routine.

myvendorname: blenderd

data[0] = 6c116a9b
data[1] = adf8a253

keys[0] = c450f9f4
keys[1] = 4d12be88
keys[2] = f52bcf4d
keys[3] = 3309994c

Decrypt xor value: 37a2cfa5

seed1: ae37b151
seed2: 6fde7999

As Dan has mentioned in his essay, the important thing to observe is that the decrypting function reveals the seeds which are encoded in the l_n36_buf function. The hiding methods that are used are quite good, but there are some weaknesses that can be exploited. Both seed1 and seed2 are xored with the same values, so it may be possible to launch a brute force attack using 2**32 different values. If we are given seed1^secretval and seed2^secretval we can cycle through the 2**32 values of secretval in search of the key. It is generally easier to simply dig the seeds out of the target.

An interesting point is that the vendor name isn't used at all in the key generation process - if the license data and encyption seeds are left unchanged, but the vendor name in the license and the vendor keys are changed, the same license key results. This agrees with Dan's findings. I have tried using quite different vendor keys, and the generated keys are the same, and only depend on feature/hostid information and the encryption seeds.

Final Notes

This pretty much sums up the new protection. Later versions of FLEXlm never call the lm_new decoding routine with a null pointer, so you never see plaintext seeds from this routine.