Welcome to the new Woodmann RCE Messageboards Regroupment
Please be patient while the rest of the site is restored.

To all Members of the old RCE Forums:
In order to log in, it will be necessary to reset your forum login password ("I forgot my password") using the original email address you registered with. You will be sent an email with a link to reset your password for that member account.

The old vBulletin forum was converted to phpBB format, requiring the passwords to be reset. If this is a problem for some because of a forgotten email address, please feel free to re-register with a new username. We are happy to welcome old and new members back to the forums! Thanks.

All new accounts are manually activated before you can post. Any questions can be PM'ed to Kayaker.

NTFS MFT Internals

Interesting low-level stuff, operating system related issues, packer/vx acrobatics, drivers and non-newbie programming in general, including win32 assembly and whatever else.
WaxfordSqueers
Senior Member
Posts: 1001
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

blabberer wrote:so it is hooking 0x70 out of 0x11c functions
That's the problem with most antivirus apps, they follow you around like a rent-a-cop tailing a teenager in a department store and interfere everywhere they can. It's a small wonder any modern OS operates at all.

When I was tracing the NTFS on a VM I found it easier to unload the antivirus and stay offline. But then I had to worry about getting caught up in the VM code. Or getting shanghied by a WaitForSingleObject type trap.

I still don't know whether to trace through such an event or try jumping over it.
WaxfordSqueers
Senior Member
Posts: 1001
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

blabberer wrote:i still dont believe shell and ole ole plays significant role in the process the object is created in R0 with ObCreateObjName & sisters and the handle is created with obpCreateObHandle and brothers does shell and ole send some undocumented DeviceIoControl to ntfs.sys
I know this is getting dated Blabs but I was thinking of opening up my investigation again, albeit temporarily, and I was reviewing the structures I know I'll encounter if I BPX on a file in explorer. Came across this concise but dated explanation of the process:

http://netez.com/2xExplorer/shellFAQ/bg_shell.html

The author explains that parsing a file path had to be extended due to the inclusion of objects that were not file objects, like printers, URLs, and virtual objects like the Desktop, Recycle Bin, etc. For that reason, different objects were given the ability to be opened by conventional means in a file manager. The means of opening them, executing them, printing them, etc., are referred to as verbs, which are defined in the registry. Also, you will find verbs as a parameter in ShellExecuteEx, the main function for processing the paths. As you trace through the code, you encounter Urlmon, which is related to processing URLs.

Of course, when you try to trace through this stuff, it is not going to process just the stuff you want, like opening an executable file. It seems to check and test for every conceivable object that can be opened by a file manager.

In the article, they stick to the convention of referring to an Item ID List, an IDL, as a PIDL, which is totally screwed. Even Microsoft does it. A PIDL is a POINTER to an IDL and a pointer is an address, not a list. To me, it is exceedingly sloppy to use the acronym PIDL in reference to an IDL, especially when you have already claimed that an item ID list is an IDL. A PIDL should be the location at which the IDL is found, not the list itself.

Although Shell and Ole may not play direct roles in the process, you have to get through their code unless you know where to go on the other side. I have tried jumping over the code to reach functions in NTFS.sys only to find the file has already been opened.

I am guessing that somewhere in an IDL, there is a reference to the MFT as an object. In fact, I seem to recall such a reference. Perhaps the MFT is being accessed early in the process as a file object, via Shell and Ole long before NTFS.sys comes into it.

Also, as I have pointed out in the past, there are intervening processes, depending on whether the file has already been moved to a cache. That includes the MFT or parts of it. During initial access to the disk volume, the MFT is moved to a filecache. Of course, it doesn't seem to move the entire MFT to a cache, so even if you end up in a cache, eventually it should go back to disk in order to refill the cache.

To complicate matters even more, Windows has different levels of caching, like WriteBehind caches. It also has search indexing. NTFS also writes to logfiles that are used by Chkdsk to rebuild the MFT should a power fail or crash happen before data from the WriteBehind cache can be written to disk. That's why I turn of that feature on a hard drive, especially when I am tracing.

I have already experienced a BSOD while tracing after which my system would not boot. However, running Chkdsk restored the system.

It's little wonder that mere mortals can make head or tail of the NTFS, especially in the deep code woods.
blabberer
Senior Member
Posts: 1535
Joined: Wed Dec 08, 2004 11:12 am

Post by blabberer »

@waxf
wasnt around so couldnt reply early tx for the link let me check it.

and if you want a collaborative trial setup a vm with say xp-sp3 and hook up windbg to it.
that way we can both go through the same path and see the same things albeit with different eyes. and others may join in too because of reproducibility in thier end at leisure

suppose you want to trace file creation through ntfs of a file name willy.billy.magicext
we can start from fopen(willybilly.magicext,"wb") or rather oldschoolish copy con blah_blah.foo ctrl+z or go with the youth of today by
'loo'sing shitstem.dognasties
WaxfordSqueers
Senior Member
Posts: 1001
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

blabberer wrote:and if you want a collaborative trial setup a vm with say xp-sp3 and hook up windbg to it.
Just so I don't have to re-invent the wheel:

1)am I hooking windbg up so it runs in XP3 or in the VM?
2)If I get a pipe set up between XP3 and VM, how am I initiating the app...say notepad? Do I start the app in windbg?

I think it's better to use an exe file because other apps like a txt file will require opening up an intermediate app to run it.

I had the basics of windbg down a year ago but i have not tried it as of late and rust has set in.

We could start with a BP on fopen but I'm pretty sure the app will be open by then.
WaxfordSqueers
Senior Member
Posts: 1001
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

blabberer wrote:... hook up windbg to it.
another question...I have configured the VM as \\.\pipe\com_1, This end is server, The other end is an application (should that be a virtual machine??) and Yield CPU on poll.

I set the boot.ini file up in the VM version of XP with "Debug" /fastdetect /debugport=com1 /baudrate=115200

When I boot into the VM now using the Debug selection in the boot menu, the mouse is acting up. It stays as an hour glass then disappears.

Have a feeling the mouse is using com1, even though it is a USB mouse. I wonder if the boot.ini should be changed to debugport=com_1 or the VM setting should be changed to \\.\pipe\com1
User avatar
Kayaker
Posts: 4169
Joined: Thu Oct 26, 2000 11:00 am

Post by Kayaker »

Do yourself a favor W and use VirtualKD to set it all up. I just reset everything up on my x64 system -> XP VMWare and it took like 2 seconds.

I forget how to start a user app remotely though, this did nothing

>.create c:\windows\system32\notepad.exe
Create will proceed with next execution
> g
blabberer
Senior Member
Posts: 1535
Joined: Wed Dec 08, 2004 11:12 am

Post by blabberer »

boot . ini in vm os /debug /debugport=com1 /baudrate=XXXX

virtual maching settings
remove the printer service which normally hogs com port 1 in wmware iirc (atleast i remember vmware player 4.01 hogging serial port 1 )
add serial port using named pipe leave all the defaults iirc \\.\pipe\com_1 select this end is server / other end is appllication

i dont know what differnce it makes by ticking yield cpu (it is not ticked by default and i leave it at default)

open windbg in host with

windbg.exe -k com :p ipe,port=\\.\pipe\com_1,resets=0,reconnect


you should have a valid connection ( i dont remember any problems with mouses disappearing probably the clock struck one and hickory dickory dock ran down
WaxfordSqueers
Senior Member
Posts: 1001
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

Kayaker wrote:Do yourself a favor W and use VirtualKD to set it all up.
Thanks for tip Kayaker. After spending multi hours trying the conventional approach, and getting nowhere, I tried Virtual KD. No better luck. I'm beginning to feel like Eagle-eyed Fleagle, walking around with a perpetual black cloud over my head. That might be a step up from Fearless Fosdick, who usually had a cannonball hole in his chest, but it didn't seem to bother him.

I got everything set up ok but when my vm boots, it freezes immediately rather than at the windows icon they suggest. Here's the virtialKD log:

*******************************************************************************
*VirtualKD patcher DLL successfully loaded. Patching the GuestRPC mechanism...*
*******************************************************************************
Searching patch database for information about current executable...
No information found.
Waiting for VMWare to initialize (5954 ms more to wait)
Analyzing VMWARE-VMX executable...
Building list of EXE sections... 11687K of data found.
Scanning for RPC command name strings...
Finished scanning. Found 41 strings.
Searching for string references...
Found 27 string references.
Found 1 structures resemblant to RPC dispatcher table.
(address 00B0EBE0, matched pointers: 27)
Analyzing potential RPC dispatcher tables...
Potential RPC table analyzis complete. Found 1 candidates.
(address 00B0E640, entries: 74, free entries: 7)
Using RPC dispatcher table at 0x00B0E640 (74 entries)
Waiting for RPC table to be initialized by VMWare...
RPC table initialized. Patching it...
Successfully patched entry #1
VMWare reset monitor activated...

and here's the windbg log:

Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Opened \\.\pipe\kd_Windows_XP_Professional
Waiting to reconnect...
Connected to Windows XP 2600 x86 compatible target at (Thu Mar 20 04:59:36.562 2014 (GMT-8)), ptr64 FALSE
Kernel Debugger connection established. (Initial Breakpoint requested)
Symbol search path is: f:\winxp\symbols;srv*f:\winxp\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows XP Kernel Version 2600 UP Free x86 compatible
Built by: 2600.xpsp.080413-2111
Machine Name:
Kernel base = 0x804d7000 PsLoadedModuleList = 0x8055b1c0
System Uptime: not available
Break instruction exception - code 80000003 (first chance)
*******************************************************************************
* *
* You are seeing this message because you pressed either *
* CTRL+C (if you run kd.exe) or, *
* CTRL+BREAK (if you run WinDBG), *
* on your debugger machine's keyboard. *
* *
* THIS IS NOT A BUG OR A SYSTEM CRASH *
* *
* If you did not intend to break into the debugger, press the "g" key, then *
* press the "Enter" key now. This message might immediately reappear. If it *
* does, press "g" and "Enter" again. *
* *
*******************************************************************************
nt!RtlpBreakWithStatusInstruction:
804e3592 cc int 3

I did not press CTRL -C or CTRL + Break.

I ran the app mentioned in the instructions inside the VM and they indicated everything was ok. Then I shut the VM down and booted the Virtual machine Monitor from the host. I tried various ways of starting the vm again but none of them worked. I can see a 'yes' under OS and Debugger but the vm just freezes.
WaxfordSqueers
Senior Member
Posts: 1001
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

blabberer wrote:virtual maching settings
remove the printer service which normally hogs com port 1 in wmware iirc (atleast i remember vmware player 4.01 hogging serial port 1 )
add serial port using named pipe leave all the defaults iirc \\.\pipe\com_1 select this end is server / other end is appllication
thanks for help, blabs. I removed the printer service, no dice. If the player was an issue, I don't think the vm would boot with a regular start up never mind debug. I'll keep it in mind.

The thing that really bugs me is setting com1 (serial0) on a pipe then setting the boot.ini file to com1 for debugport. If windbg is using com1 for a pipe how does windoze in the vm use it? I tried setting it to com2 and I even configured the vm for 4 serial ports but no luck. I say that because com1 on the vm is supposed to be mapped to the com1 irq/mem address of the host.

Somehow the mouse is getting onto com1 even though it's on a USB port on the host machine. It looks to me like an old-style IRQ conflict but I don't see how the mouse is getting onto irq 4 where com1 is located.

Mind you, the vm insists on loading both the PS/2 mouse and the HID variety via USB. It wont let me uninstal the ps/2 driver.

Part of the problem may be that my Intel mobo only supplies one com port.
User avatar
Kayaker
Posts: 4169
Joined: Thu Oct 26, 2000 11:00 am

Post by Kayaker »

Hmm, it sounds like you were 100% successful. That's exactly what I get. Now you should be under Windbg control but need to press F5 (g) to allow the VM to continue loading. Soon you should be able to interact with the VM. Back in Windbg, issue Debug/Break and the VM will "freeze" again but you can use Windbg commands, i.e. !process 0 0 should give a list of all VM processes running.
WaxfordSqueers
Senior Member
Posts: 1001
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

Kayaker wrote:Now you should be under Windbg control but need to press F5 (g)....Back in Windbg, issue Debug/Break and the VM will "freeze" again but you can use Windbg commands, i.e. !process 0 0 should give a list of all VM processes running.
Sheer genius, kayaker...as usual. No one mentioned F5 and I'm too stupid to remember what blabs told me about it.

Did the F5 and the vm continued loading. Waited till it was fully booted and windbg changed from busy. Did a cntl - break but did not notice anything. When I did the !process 0 0 it gave me the list.

The vm just sits there, however, with an hour glass where the mouse cursor is. If I move it, the cursor disappears.

I was at the same point with the old method so maybe all I had to do was boot windbg and follow the same procedure. I'll try it.

I had the notion that I needed to use the vm. I presume I go into windbg and issue an attach to process but I don't have notepad running in the vm and there's no way to start it till I sort out the mouse issue.

Need to spend more time with it.
blabberer
Senior Member
Posts: 1535
Joined: Wed Dec 08, 2004 11:12 am

Post by blabberer »

f5 = g == go same from good old dos days old debug.com uses g = go

Code: Select all


C:\WINDOWS\system32>debug command.com
-g
Microsoft(R) Windows DOS
(C)Copyright Microsoft Corp 1990-2001.

C:\WINDOWS\SYSTEM32>
windbg printed that for you when it broke and asked you to do it :) if you didnt intend to break didnt it ?

anyway now that you are into windbg after you pressed f5 or typed g and the vm booted up fully

if you intend to break again press ctrl+break

when broken when you want to run the target again type g and hit enter or press f5

also note what you are now into is kernel debugging not your usual user mode debugging

you do not attach to process from here (only remote debugging will let you either create or attach to process and that is a different beast

i noticed kayaker doing .create

no .create or .attach wont work in this setup
( this needs -premote option with debuging tools runnning on both machines one as a server and other as a client or a dbgsrv.exe or remote.exe in commandline )

in this setup you need to set breakpoints and start the process in vm for windbg to break on process creation

for example

do ctrl+break in windbg

type bp nt!NtCreateProcessEx ; g and hit enter

now in vm click notepad.exe and windbg should break again
WaxfordSqueers
Senior Member
Posts: 1001
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

blabberer wrote:windbg printed that for you when it broke and asked you to do it :) if you didnt intend to break didnt it ?
mighty decent of it, I'd say.
blabberer wrote:now in vm click notepad.exe and windbg should break again
I can't get into vm and I have to find out why. I had the same issue with softice if I tried to start in debug mode. Don't ask why I wanted to do that...just curious.

I have several things to investigate. For one, my bios can be updated and there may be things in there that are affecting the overall operation. Right now, I am trying to foolproof the bios update using a USB thumb drive but that's another experience by itself.
blabberer
Senior Member
Posts: 1535
Joined: Wed Dec 08, 2004 11:12 am

Post by blabberer »

I had the same issue with softice if I tried to start in debug mode.
do i infer that you have softice installed in the vm ? and you have connected teh vm to windbg too ?

if so ctrl+break will be caught by softice

you either need a neat and clean vm or you have to muck with softice i3here off
WaxfordSqueers
Senior Member
Posts: 1001
Joined: Tue Apr 06, 2004 11:00 am

Post by WaxfordSqueers »

blabberer wrote:do i infer that you have softice installed in the vm
I don't run softice from boot, I start it once I get to the desktop. I double-click it to run it and it goes through it's initialization and loads all it's crap. It's driver is loaded and that may be creating problems. Kayaker would know more.

I don't think it would be trapping anything but I could clone my present system and remove SI. Driverstudio is another matter. I have never used it and have no idea what its drivers are up to.

My problems are occurring before windbg is started. Before I loaded the app Kayaker suggested I was trying to launch kernel debug mode and that's when the mouse started to freeze. I thought it might be due to the statements required in the vmx file to let SI run but defeating them did nothing.

I think my problem is a conflict between the mouse and com1 but I will definitely look into the SI matter.

Thanks.
Locked