Immortal Descendants Proudly Presents

08 January 2000

Advanced FLEXlm Tactics

Defeating vendor defined encryption routines.
Difficultly Level:

( ) Beginner - ( ) Easy - (X) Medium - () Advanced
All Rights Reserved Amante4 ID 2000

1. Introduction

There have been a fair number of essays on this topic in the past. Many of them I've learned from a great deal. These essays have shown us how to sniff out the vendor code structures of a target, get VENDORKEY5, decrypt the encryption seeds now having VENDORKEY5, and finally making a license for our target. These methods work for a large amount of FLEXlm protected targets.

FLEXlm provides alternative ways for a customer to customize the way they implement FLEXlm on their targets. One such customization is the addition of a vendor defined encryption routine. This essay is intended to show what this added protection is about and how to defeat it with ease.

2. What You Need

Target :

Tools :

Knowledge :

3. The Essay

One day I was trying to fish out the vendorcode structures out of a target I use on a regular basis. I broke on the lc_init function and looked at the parameters passed into the function in the way I normally do. I saw something strange, that at first I could not explain. As we know from previous learning, one of the parameters passed into the lc_init function is the VENDORCODE structure. The structure looks like so :

typedef struct vendorcode5 {
  short type; /* Type of structure */
  unsigned long data[2]; /* 64-bit code */
  unsigned long keys[4];
  short FLEXlm_version;
  short FLEXlm_revision;
  char FLEXlm_patch[2];
  char behavior_ver[LM_MAX_BEH_VER + 1];

#define LM_MAX_BEH_VER "06.0"

In the above structure, data[0] = encryption seed 1 XORed with vendor key 5, data[1] = encryption seed 2 XORed with vendor key 5 keys[0..3] = vendor keys 1 to 4, behavior_ver[] = string containing FLEXlm version ( in this case = "06.0").

The odd thing I saw was that data[0] was the same value as data[1]. Using those values and the values for keys 1 through 4 I found VENDORKEY5. Then, in the traditional way I XOR'ed VNEDORKEY5 with the encrypted seeds and the result was 0 for both clear seeds. I immediately knew something was fishy. Keep in mind that this probably is not the case for your target. I think it was just the way this vendor chose to define them. But it gave me a clue that something else was happening. The clue, in general, for any target that is using such a trick, is when the conventional methods of generating a license for the target fails because the license is invalid.

OK, so now after reading some FLEXlm documentation, I found the following :-

The FLEXible API allows you to control the licensing behavior of your application with a set of attributes. FLEXlm attributes allow you control over licensing policy, internal operations of FLEXlm (e.g. use of timers, etc), and control of the licensing parameters of your process (e.g., define how FLEXlm will define `username', `hostname', and `display name,' etc. for managed license distribution.). To set FLEXlm attributes, use the lc_set_attr call.

We now have a function to have a look at. According to the documentation, here's the prototype for the lc_set_attr function :-


status = lc_set_attr(job, key, (LM_A_VAL_TYPE)value)


Sets a FLEXlm attribute. The key describes which attribute to set, and the value is the value for the attribute. See the header file lm_attr.h for key constants and value types. Parameters :-

(LM_HANDLE *) job
From lc_init().
(int) key
Which attribute to set.
(LM_A_VAL_TYPE) value

Value to set it to. Values should be of the appropriate type for the particular attribute (see lm_attr.h), but should be cast to LM_A_VAL_TYPE. Return :-

(int) status
0 = OK, !=0, error.

From this function we can see it sets a particular key to a value. We now need a key that would cause FLEXlm to use a vendor defined encryption routine. Searching the documentation once again I found the following key :-



Pointer to a function returning (char *). Return value is the license key. The function pointer crypt can be set to point to a vendor-supplied authentication routine to be used in place of the default routine.


FLEXlm standard authentication routine. This key will do the trick!. So the target must use the lc_set_attr function to set LM_A_USER_CRYPT to point to the function that performs the encryption. Another thing to note here is that the return value is a string which is the license key. This is too good to be true. Remember that the function looks like this: lc_set_attr(job, key, (LM_A_VAL_TYPE)value). The key to set is LM_A_USER_CRYPT and looking in lm_attr.h we see the following :-

#define LM_A_USER_CRYPT        15     /* PTR to func returning (char *) */

So we now know that 15 (or 0xF in hex) should be pushed on the stack just prior to a lc_set_attr call in our target. Let's disassemble our target and look at the calls to lc_set_attr.

:004011DD PUSH 004011B0 ; value = 0x004011B0 --> This is where the vendor defined encrytion routine resides
:004011E2 PUSH 0000000F ; LM_A_USER_CRYPT
:004011E4 PUSH ESI ; job

* Reference To: LMGR326A.lc_set_attr, Ord:003Dh

:004011E5 CALL 00401222 ; call function

Now before we go on, lets create a dummy license file (license.dat) and put it in the same directory as the target exe. mine looked something like this:

FEATURE f1 blenderd 1.0 permanent uncounted 000000000000 HOSTID=ANY

You should know how to sniff out the feature name of "f1" from previous essays. Also note here that I've chosen to use the infamous Nolan Blender daemon and keys for this example. I've also zero'ed out the encrypted license key and thus thats why I call it a dummy license file. Here are the keys I used to create the target application so that you can try to generate licenses for the target the usual way to verify they won't be valid.

#define ENCRYPTION_SEED1 0xae37b151
#define ENCRYPTION_SEED2 0x6fde7999
version 6 sdk keys to be placed in lm_code.h
VENDOR_NAME blenderd
VENDOR_KEY1 0xc450f9f4
VENDOR_KEY2 0x4d12be88
VENDOR_KEY3 0xf52bcf4d
VENDOR_KEY4 0x3309994c
VENDOR_KEY5 0xaefa9027

Now load up SoftICE and set a breakpoint on 0x004011B0. This is the vendor defined encryption routine we found in the above call to lc_set_attr.

Get out of SoftICE with F5. The program will ask which feature to checkout with a default of "f1". Just hit return for the default and SoftICE should fire. You're now sitting at the start of the routine. If you recall this function returns a string (char *) that is the encrypted license key. So at this point simply hit F12 to return from the function. The return value is sitting in EAX, so do a d eax in SoftICE and look at the data window. You should see the following string in the ASCII part of the data window: "123456789123". This is the encrytion key to use in place of the zero'ed out one in our dummy license file. So lets try it. My license file now looks like this :-

FEATURE f1 blenderd 1.0 permanent uncounted 123456789123 HOSTID=ANY

Run the program again and you should see a message saying the feature was checked out successfully. I've supplied the source code for the sample target I've provided (it's just a modified sample that comes with the FLEXlm SDK) As you can see, the vendor defined encryption routine I wrote is very simple. It just passes back a fixed string. I used this for simplicity, the point to be noted is that no matter how complicated a vendor makes this routine the simple fact remains that the key is returned from this function as plain text in memory.

In summary, I've been able to use the following steps to quickly and easily generate the license keys for any feature that is needed.

1). Find encryption routine by snooping the lc_set_attr call.
2). Put breaks just prior to the lc_checkout call and at the encryption routine found above.
3). Put dummy feature lines in the license file for all needed features. These lines should have all valid info except for the license key.
4). On break to lc_checkout, change the feature name passed to lc_checkout in memory to the desired feature name.
5). You should break on the encryption routine next.
6). Return from routine and look at return value (license key for feature).
7). Replace this value into the license file.
8). Repeat steps 4 through 7 as many times as is necessary.

Here's the code listing of the target lmflex.c :-

/* An Example application that uses a vendor defined encryption routine */

/* Modified from the SDK version of lmflex.c by: Amante4  1/8/00 */

#include <stdio.h>
#if (defined( __STDC__) || defined(_WINDOWS)) && !defined(apollo)
#include <stdlib.h>
#include <time.h>
#include "lmclient.h"
#include "lm_code.h"
#include "lm_attr.h"
#define LICPATH "license.dat;."

#define FEATURE "f1"

char feature[MAX_FEATURE_LEN+1];
LM_HANDLE *lm_job;

char *my_crypt(LM_HANDLE *, CONFIG *, char *, VENDORCODE *); // prototype for my encryption function

    if (lc_init((LM_HANDLE *)0, VENDOR_NAME, &code, &lm_job))
        lc_perror(lm_job, "lc_init failed");

    lc_set_attr(lm_job, LM_A_USER_CRYPT,(LM_A_VAL_TYPE)my_crypt); // Tell FLEXlm I'm using a vendor defined routine

    printf("Enter feature to checkout [default: \"%s\"]: ", FEATURE);

    fgets(feature, MAX_FEATURE_LEN, stdin);
    feature[strlen(feature)-1] = '\0'; /* truncate trailing newline */
    if (!*feature) strcpy(feature, FEATURE);

    lc_set_attr(lm_job, LM_A_LICENSE_DEFAULT, (LM_A_VAL_TYPE)LICPATH);
    if (lc_checkout(lm_job, feature, "1.0", 1, LM_CO_NOWAIT, &code,
        printf("checkout failed... press return to exit...\n");
        lc_perror(lm_job, "checkout failed");
    printf("%s checked return to exit...", feature);
*    Wait till user hits return
*    getchar may be interrupted by SIGALRM, so we loop if necessary
    lc_checkin(lm_job, feature ,0);


/* My vendor defined encryption routine */
char *my_crypt(LM_HANDLE *job, CONFIG *conf, char *sdate, VENDORCODE *key)
    char *lic_key;
    char *modify_key(char *); // prototype

    lc_set_attr(job, LM_A_USER_CRYPT, (LM_A_VAL_TYPE)0);
    lic_key = lc_crypt(job, conf, sdate, key); // This is the normal way to get the license key
    lc_set_attr(job, LM_A_USER_CRYPT,(LM_A_VAL_TYPE)my_crypt);

/* modify the license-key */


char *modify_key(char *key)
    return ("123456789123"); // just return a constant string
    // This function could contain a really complicated algorithm
    // to change the normal license key in some way
    // but who cares.

4. Final Notes

Once again FLEXlm proves to be easily defeated. The customization mentioned in this essay prevents a generic crack of FLEXlm if the vendor impements it. This will not stop a dedicated cracker from busting the protection as seen above. This is not the end of FLEXlm reversing by far. There's still many more reversing sessions on this popular commercial protection scheme. I also believe that the methods used here can be applied to also find vendor defined string filters. An interesting exercise would be to extract out the vendor defined encryption routine from the target and incoorperate it in the license generation programs of FLEXlm. That way you could have a program to generate the licenses automatically.

later, Amante4.

5. Legal Notes

Remember that you can do all the things written here only at YOUR risk. I do NOT take any liability for YOUR acts.