View Full Version : Crash analysis of a plugin in a browser [need help]

October 7th, 2009, 13:10
| 0 - Introduction |

This is a request for help of a crash analysis of xxxxx10b.ocx crashed within a browser. The crash dump is analyzed but with 2 mysteries unexplained.

There are 4 sections below:

1. Crash dump analysis
The crash dump of the browser was analyzed with windbg. This section includes the result of the !analyze -v command, the r(register) command and the page protection information. Comments are colored in blue.

2. IDA analysis
The binary of xxxxx10b.ocx is loaded into IDA for static analysis. This section contains the disassembly of the caller before the crash. The nProcessor variable was traced by cross reference and the purpose is found to be storing the number of processors.

3. Disassembly around crash point
Since the crash point != eax, the last function pointer to call, the disassembly starting from eax are analyzed to help finding the most reasnoable execution path.

4. Analysis evaluation
The result and analysis of the incident is presented in this section. The unknowns / questions are also written here.

| 1 - Crash dump !analyze -v |

EXCEPTION_RECORD: ffffffff -- (.exr ffffffffffffffff)
ExceptionAddress: 035db7c3
ExceptionCode: c0000096
ExceptionFlags: 00000000
NumberParameters: 0

BUGCHECK_STR: c0000096


PROCESS_NAME: browser.exe

ERROR_CODE: (NTSTATUS) 0xc0000096 - {

LAST_CONTROL_TRANSFER: from 045761a0 to 035db7c3

WARNING: Frame IP not in any known module. Following frames may be wrong.
0617e334 045761a0 00000001 18b83ef0 00000000 0x35db7c3
0790d444 00000000 00000003 00080813 00000000 Xxxxx10b+0x761a0
As seen from the stack, the return address is 45761a0, arg0 is 00000001. Matched push 0x1; call eax at .text:1007619C.
18b83ef0 matched push edi at .text:10076170.

045761a0 8b4f44 mov ecx,[edi+0x44]



SYMBOL_NAME: Xxxxx10b+761a0


IMAGE_NAME: Xxxxx10b.ocx


STACK_COMMAND: .ecxr ; kb

FAILURE_BUCKET_ID: c0000096_Xxxxx10b+761a0

BUCKET_ID: c0000096_Xxxxx10b+761a0

Followup: MachineOwner

0:029> r
eax=035db7b0 ebx=00000000 ecx=00020002 edx=08740000 esi=00000002 edi=18b83ef0
eip=035db7c3 esp=0617e338 ebp=0790d444 iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
035db7c3 ed in eax,dx

0:029> !vprot eax
BaseAddress: 03590000
AllocationBase: 03590000
AllocationProtect: 00000004 PAGE_READWRITE
RegionSize: 00068000
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
There is no EXECUTE access in this memory region.

0:029> !vprot 45761a0
BaseAddress: 04501000
AllocationBase: 04500000
AllocationProtect: 00000080 PAGE_EXECUTE_WRITECOPY
RegionSize: 002e9000
State: 00001000 MEM_COMMIT
Protect: 00000010 PAGE_EXECUTE
Type: 01000000 MEM_IMAGE
While the return address has EXECUTE access.

| 2 - IDA |

.text:10076170 sub_10076170 proc near ; CODE XREF: sub_10082F10+131j
.text:10076170 ; sub_10088C30+251p
.text:10076170 57 push edi ;
.text:10076171 8B F9 mov edi, ecx
.text:10076173 83 7F 44+ cmp dword ptr [edi+44h], 0
.text:10076177 74 3B jz short loc_100761B4
.text:10076179 56 push esi
.text:1007617A 33 F6 xor esi, esi
.text:1007617C 39 35 8C+ cmp nProcessor, esi
.text:10076182 7E 2F jle short loc_100761B3
.text:10076184 loc_10076184: ; CODE XREF: sub_10076170+41j
.text:10076184 8B 47 44 mov eax, [edi+44h] ; edi+44 = 18b83f34 -> 1886bc78
.text:10076187 83 3C B0+ cmp dword ptr [eax+esi*4], 0
.text:1007618B 8D 04 B0 lea eax, [eax+esi*4] ; for esi=2, [eax+esi*4] -> 00020002
.text:1007618E 74 1A jz short loc_100761AA
.text:10076190 8B 00 mov eax, [eax]
.text:10076192 85 C0 test eax, eax
.text:10076194 74 0A jz short loc_100761A0
.text:10076196 8B 10 mov edx, [eax] ; edx = 08740000
.text:10076198 8B C8 mov ecx, eax ; ecx = 00020002
.text:1007619A 8B 02 mov eax, [edx] ; eax = 035db7b0
.text:1007619C 6A 01 push 1
.text:1007619E FF D0 call eax ; crash later at 035db7c3
.text:100761A0 loc_100761A0: ; CODE XREF: sub_10076170+24j
.text:100761A0 8B 4F 44 mov ecx, [edi+44h]
.text:100761A3 C7 04 B1+ mov dword ptr [ecx+esi*4], 0
.text:100761AA loc_100761AA: ; CODE XREF: sub_10076170+1Ej
.text:100761AA 46 inc esi
.text:100761AB 3B 35 8C+ cmp esi, nProcessor ; nProcessor is 2 in the crash dump.
.text:100761AB 40 37 10 ; cmp esi, 2
.text:100761AB ;
.text:100761B1 7C D1 jl short loc_10076184 ; Why would jl jmp when esi = 2?

| 3 - Disassembly on 035db7b0 |

Crash dump disassembly:

035db7b0 c00000 rol byte ptr [eax],0x0
035db7b3 0000 add [eax],al
035db7b5 0200 add al,[eax]
035db7b7 0000 add [eax],al
035db7b9 0100 add [eax],eax
035db7bb 0000 add [eax],al
035db7bd 0100 add [eax],eax
035db7bf 0128 add [eax],ebp
035db7c1 55 push ebp
035db7c2 e7ed out ed,eax

Reconstructed disassembly(Since the code here appear to be self-modifying):

035db7b0 0000 add [eax],al
035db7b2 0000 add [eax],al
035db7b4 0002 add [edx],al
035db7b6 0000 add [eax],al
035db7b8 0001 add [ecx],al
035db7ba 0000 add [eax],al
035db7bc 0001 add [ecx],al
035db7be 0001 add [ecx],al
035db7c0 2855e7 sub [ebp-0x19],dl
035db7c3 ed in eax,dx

| 4 - Analysis evaluation |

From the information available, the following points are guessed:

- The crash point was at 035db7c3.
- The caller was 0457619e by call eax.
- eax was 035db7b0.
- The machine code at 035db7b0 to 035db7c0 shows that the control flowed from 035db7b0 to 035db7c3. ((4 * 0xb0) & 0xff = c0 at 035db7b0)
- The register dump shows that the loop at loc_10076184 were executed before the crash.

However, there are 2 mysteries:

1. Why esi = 2 but it can still branch from 100761b1?
2. Why can 035db7b0 being executed when the protection info shows that there was no EXECUTE access?
Experiment shows that setting eip to memory location with the same characteristic as 035db7b0 will result in access violation.

Any comments or suggestions are welcome!

October 7th, 2009, 23:56
Since the default value of nProcessor is 0x0F, the best bet would be a racing condition that caused nProcessor to be set as 2 after the jl branch at .text:100761B1 occurred.

Thus allowing out of bound function pointer being de-referenced. With many lucks, the chain of pointers isn't broken in the middle and finally fired call eax.

Then why is 035db7b0 executable? How can I confirm this by examining from the minidump with all optional data included? (if !vprot is unreliable)

Should check the page table manually???

October 8th, 2009, 01:19
Hi Kayaker

I feel comfortable for your explanation to the page protection question since the code has really been executed. And the information from !vprot cannot be 100% interpreted as how the facts are.

35db7b0 is a very very very very very(5 passes!) lucky one if 0x00020002 is not really a pointer. Because on a live debug session, the ([edi+44])+8 is something like 0x000200xx and it can only be de-referenced one or twice or thrice and then it would point to ?????(unmapped?) memory instead of going to call eax.

Yup !! Google has many hits , the cases I have looked of didn't look like the accident of mine or important information are absent. And since Xxxxx is a famous crashing engine when run with incorrect script......there could be many ways to crash it so I chose to inspect the case exclusively after reading a few search results.

From your action(brute force the masks and doing some manual RE) I could see that you are very willing to help! Thank you!

October 8th, 2009, 01:27
Hi nothize

Yes, my comment about JL was garbage, which is why I deleted my post after rereading it, but obviously not before you had read it Sorry about that.

Just to repost the sensible stuff:

Two MS definitions of PAGE_READWRITE indicate that execution IS allowed (if DEP isn't enabled). DEP is always enabled in 64 bit Windows.

PAGE_READWRITE - Read, write, and execute access to the specified
region of pages is allowed. If write access to the underlying section is
allowed, then a single copy of the pages are shared. Otherwise the pages
are shared read only/copy on write.

Enables read-only or read/write access to the committed region of pages.
If Data Execution Prevention is enabled, attempting to execute code
in the committed region results in an access violation.


October 8th, 2009, 01:33
HI Kayaker

Hehe take it easy!! I have deleted the assumption of race condition for the nProcessor variable in the second thought. But now I'd rather take it back again.

And you taught me to look for the reference although the symbol is verbose enough.

Thanks again!

October 12th, 2009, 13:08
After a live debug session tonight, which set breakpoints on where writes to nProcessor, it is seen that the routine of updating this variable is often being called.

Since this variable always being updated to 0xF before setting to the number of processors, 2 in my case, and nProcessor looks like a global variable(resides in the data section), race condition is likely the culprit of the crash.


.text:100871E8 83 F8 0F cmp eax, 0Fh
.text:100871EB C7 05 8C+ mov nProcessor, 0Fh
.text:100871F5 7F 05 jg short loc_0_100871FC
.text:100871F7 A3 8C E0+ mov nProcessor, eax

And as simulated, when there are more than one instance of the browser window using the plugin, a race condition of the nProcessor can be created by freezing at 100871f5 to wait for another thread to access this variable to achieve the out of bound function pointer de-referencing.

October 12th, 2009, 23:35
That's some nice intuitive work there nothize to come up with, and test, a race condition as a possible explanation. Since we're not discussing breaking a commercial protection or anything, I think it can be said that the file in question is Flash10b.ocx.

It makes sense in retrospect, you can see how the race condition might occur, but it's amazing that it *would* occur because of the timing involved, all within the space of a couple of instructions.

What it seems to look like is this. There are actually 2 occurences of the following code (which doubles the chances of an error), where the global variable nProcessors is changed briefly to what in most systems would be an incorrect value (at least I don't know anyone with more than a Quad-core!)


:10087686 call GetNumberPhysicalProcessors
// Call GetSystemInfo, return [SystemInfo.dwNumberOfProcessors]

:1008768B cmp eax, 0Fh
:1008768E mov nProcessors, 0Fh
:10087698 jg short loc_1008769F

:1008769A mov nProcessors, eax

In psedocode:


n = GetNumberPhysicalProcessors();
nProcessors = 15;
if ( n <= 15 )
nProcessors = n;

The problem appears to be that nProcessors briefly (over the course of 2 instructions) can have the value 0x0F.

At the same time, there are 2 separate loops which might be running (in another browser instance/thread) which *uses* the nProcessors global variable. They are simple loops of the kind:


for ( i = 0; i < nProcessors; ++i )


The race condition would occur if the loop limit nProcessors was checked, at the same time that the global nProcessors variable had been changed to 0x0F.

Wow! what are the chances of that? Nothize nicely simulated the race condition. Could there be another explanation?

I suppose this is what can happen when you get too many cooks in the kitchen. There are a few other places where GetNumberPhysicalProcessors is called (though the return value isn't stored in a global variable. Programmer A figured the "upper limit" on the number of processors would be 8, in another case Programmer B chose 0x10, and Programmer C above decided on 0x0F. Hmmm...

October 13th, 2009, 01:07
Thanks Kayaker for the supplementary explanation!

The experiment mentioned in my last post was based on Flash10c.ocx thus the offset might have a bit difference. That's because the crash was dumped on another machine but the experiment was done on my own machine.

Here are the steps to re-produce the situation:

1. Open a browser(in theory, both the .ocx and .dll version have the same problem).
2. Enter an URL that has flash embedded. (make sure the flash plugin is loaded).
3. windbg -pd -pn iexplore.exe
4. Set 3 breakpoints:
i) at the first jg (after cmp, nProcessor, 0Fh)
ii) at the second jg (after cmp, nProcessor, 0Fh)
5. Start another browser instance, load another URL that has flash.
6. g (continue execution even bp 0 or 1 hit unless ~# is not showing thread 0)
7. ~f freeze the current thread that is at jg
8. g
9. Access violation........

It should be noted that during the experiment if the jg is at thread 0, no other thread will be likely trying to read from nProcessor. But when jg is at other threads, then thread 0 will try to read it within the loop.

(I'm going to try to crash firefox using the dll version, so have to submit the reply now!!)

October 13th, 2009, 02:01
I tried with Firefox but it always use thread 0 when accessing nProcessor.(read or write)

So npswf32.dll on firefox is not affected by this problem for this test case. It could be the nature of Firefox to not call the plugin from a worker thread.

Chrome should be tried too, though it's not available on this machine.....