NTFS MFT Internals

[WaxfordSqueers]

get windbg its is young creaky and cranky

Still have not attacked the theme of your post, busy reading about kernel processes. However, one of my sources uses LiveKD. My understanding, from well in the past, is that windbg requires a remote processor set to debug mode. That's why LiveKD was produced, to use windbg on one system.

What's your take on that? What kind of setup are you using?

I normally work on a laptop these days, running win 7 but I have a desktop nearby running XP, with all my reversing stuff on it. I think I have windgb on there as well. I have a primitive network setup between the two but I use it primarily for file sharing. Could I setup windbg on the laptop and debug on the desktop. Reason I ask is that the laptop is online and has access to symbols but the desktop does not have access. I could always d/l the symbols if packages are still available.

[WaxfordSqueers]

...busy reading about kernel processes.

Sorry...had to take a break from reading and express my frustration. Windows is bloatware of the highest order. I am reading about objects and still trying to figure out why this obfuscation is necessary. I am reading it only because I have come across a reference to a hardware object in the code (I call it a disk drives, but what would I know, being one of those lowly hardware types?).

I imagine the people who came up with this have pants that come 6" above their shoe tops, held up with suspenders, belts, and ropes for good measure. They must hate hardware because they have taken great panes to make it appear as if their obfuscated, virtual reality is somehow running the show while the processor is there as a show piece, like a blond babe with a sugar daddy. Object-oriented logic is the quantum physics of computing. As Feynman claimed, it works, but no one knows why.

I swear, they could cut out 3/4 of windows overhead and it would still work fine, even better. Is it just me who has noticed while tracing through windows code, how many times a piece of code is repeated, like when verifying a string? It's even worse in the kernel, where myriads of bloated code seems to run endlessly in circles. I don't know how many times I have seen windows code go over the same function several times for no apparent reason. I wonder if Windows programmers have taken the time to run through their bloatware at a low level.

I'd be willing to bet they have solved problems where the code has gotten away from them by patching it so it would return to the calling function.

Gripe ends. Sorry Woody, for wasting your disk space. Move this to off-topic, if you like, or just delete it.

[Aimless]

Sorry...had to take a break from reading and express my frustration too!!

But unfortunately, it's linux this time. :P

Last time I tried Linux (Debian, Red Hat, Slackware, Ubuntu, Mint, Puppy, Parsix, Cent and Gentoo), I came away feeling dirtier than when I had used windows. Linux is not really what it's made out to be. It's one of the least robust OSes I've ever worked on. It's much-vaunted stability, is in reality, about as robust as a freshman girl's virginity on Prom Night with her favourite quarterback crush her current date and lots of empty bedrooms available with free condoms strewn around. The only good system was the UNIX one (AIX and System V) and occassionally, Solaris (but only when running on Sparcs). Netware had uptimes in YEARS, not months, but was too exclusive. FreeBSD was okay - strictly when running it as a web server, not otherwise. No doubt Linux developers often advertise it as the best web server, or file server, or print server --- never a good desktop, though.

The ONLY other system I have come across to liking currently is MAC OS X Lion (but the mountain lion version squeals like a cat when it's tail is stepped on)

I've got XP Pro with SP3 running on my many systems and it's NOT had a downtime for the last 8 months (yeah, that's not a typo) and I've yet to see a memory leak or slowdown. I have a Windows 7 at work and I'm throwing everything on it including the kitchen sink and I've YET to re-install or repair it in the last 3 years (yep, not a typo either).

Just thinking, maybe if linux supported ALL hardware and peripherals that Windows does, had all the backward compatibility that windows has, ran on so many types of PCs, Netbooks, Laptops, Tablets that windows does, came across so many malware that windows did, had to cater to so many different regional settings, had to work for so many varied industries in the world, the code would, perhaps, have not be that bloated. Or maybe if it was not backward compatible with so many software tools and older versions of the programs, it would be 'streamlined' too.

I've also seen hardened linux admins get a confused look on their face when asked why this or that hardware is not being recognized. And I'm talking about the latest version of Ubuntu and Mint -- leave alone slack or debby. Compared to which windows can be setup by a newbie too (another reason for bloated code?). And while linux gurus extoll the virtues of working in CLI (not that I mind it, I use it myself - rarely, though ), have they ever tried image manipulation in a CLI? Or music editing in a terminal window? Or, dare I say it, even DEBUGGING CODE in a CLI? Perhaps, to SUPPORT the WINDOWS based debugging, the code is bloated. Perhaps, to SUPPORT so many commands in the debugger, the code is bloated. Try running IDA in CLI. And let me know if you don't go goggle-eyed at that. GDB is good. But not THAT good. Not, OLLY good. Not, Windbg good. Not, IDA Good.

Thankfully, the people who came up with this code in windows wear pants that come 6" above their shoe tops, held up with suspenders, belts, and ropes for good measure -- because the way Linux works, I am sure THEIR developers coded it without a pair, in their underwear, in their parent's house too, while wiping the sauce from their faces, after they had played with their GI Joe figurines, that is (Hey, it could've been spongebob too, for all I know :P) .

MS Paint is not better than Adobe Photoshop, just because its more 'streamlined', is it? I mean, Photoshop is BLOATED for a reason, I think.

Gripe over. Sorry Woody, for wasting your disk space. Move this to off-topic, if you like, or just delete it.

Hey Waxy, love you. Peace, brother.

Have Phun

[WaxfordSqueers]

Last time I tried Linux (Debian, Red Hat, Slackware, Ubuntu, Mint, Puppy, Parsix, Cent and Gentoo), I came away feeling dirtier than when I had used windows.

Hey. Aimless...whazzup?

I am not complaining about Windows as an OS in general. I am venting at the infrastructure, which is based on abstraction so unrealistic that it divorces an average user from the machine. As I said, I am coming from a hardware POV which is where my expertise is based. Windows marketers have passed the system off as a glossy front end and the developers have pushed concepts like wizards. The wizards are so stupid, that if inf files are missing from the INF directory, they are absolutely useless. I have fixed many a problem by hiding an INF file and forcing Windoze to load 'MY' driver. Even at that, it whines at you.

Also, anyone who has tried to use the Windows help system has surely cried out in frustration. They lead you through questions which in my experience end with them telling you they don't have the answer.

I stay away from Linux because of it's command line system, which is archaic. I started out in computers in the late 1970s and the kind of mickey mouse utilities you still find in the Linux command line system were out-dated in the 1980s. Although you can apparently do more with it than DOS, I found DOS easy to use and logical. There's nothing logical to me about files with no extensions and directories and disk drives regarded as files. And the security is just plain annoying. I don't need it and it is a hassle trying to read and set permissions on everything encountered.

The Linux GUI is not bad so long as you don't have to look under the hood. I preferred KDE to Gnome but I could not see anything offered that I did not have in Windows. I use both Win 7 and XP and put up with the arrogance built into either system whereby Msoft treats the user as a noob who has never run a computer. Being told that I don't have permission to use a file or directory is way beyond cool. I fix it all with my 'take ownership' utility, as I spit. "ef you, Gates".

I have never consider MAC simply because I'd have to change all my apps.

[blabberer]

what is your take on this

sysinternals livekd and latest windbgs local kernel debugging both are fake those are not debugging the system live.
they create a snapshot and provide various facets about the kernel state.
(yes it is quiet usefull under many circumstances .
I use the facility quiet often
have written a plugin for ollydbg to use the functionality inside ollydbg blah blah blah blah )
but both of them are not kernel debugging they are similar to
MRI or CAT or Sonograph or Endoscope seeing something without actually seeing it
hearing something without actually hearing and
feeling something without actually feeling them

yes you need two machines and a transport between them for windbg to work if you want to work with real hot iron

you can kinda fake this and perform a single machine kernel debugging if you don't have to work with hot iron

simply get and install one of the virtual (PC / ware / box / .....) (hereinafter called as target) in your laptop (hereinafter called as host)
install your favorite OS (preferably xp sp3 to start with) in the target
create a virtual transport using named-pipe between host and the target

open windbg in host and connect it the virtual transport send a break-in from host (ctrl+break) or from target (ctrl+alt+fn+prtscreen)

enjoy uninterrupted kdculation

Zw versus Nt

read the articles they both exist for a reason bloat with garter-belt is not one of them

http://msdn.microsoft.com/en-in/library/windows/hardware/ff565438%28v=vs.85%29.aspx
http://www.osronline.com/article.cfm?id=257

should x api be called before y DDI in fist pose

windbg knows ntfs better than me

started a vpc - kd session

asked what windbg knows about ntfs,mft,read,write with

kd> x ntfs!*r*mft*r*
fc34fc03 Ntfs!NtfsReadMftRecord = <no type information>
fc3a3545 Ntfs!NtfsDefragMftPriv = <no type information>
fc36d2af Ntfs!NtfsReserveMftRecord = <no type information>
fc33cd0f Ntfs!NtfsReadMftExceptionFilter = <no type information>

set a log break on one of them and on nt!NtCreateFile and dumped the stack to examine it
a bit of googling and observation shows that the third argument to this call is MFT_SEGMENT_REFERANCE or FILE_REFERANCE
again a bit of googling says ntfswalk by dmitybyrant can show the mft details downloading and running it and spleunking a little with windbg
it seems i can narrow down mft reads and ntcreatefile

the script as used for logging mft reads and create file as follows

printdec.txt (ntfswalk shows mft# in decimal so 
.printf "mft# = %6d\tFileName = %msu\n" , poi(poi(@esp+c)) , poi(poi(@esp+c)+20)+8 ;
gc 

filepattern is last posts script

as /mu ${/v:instr} @@c++((wchar_t *)(((nt!_UNICODE_STRING *)((nt!_OBJECT_ATTRIBUTES *)@@masm(poi(@esp+c)))->ObjectName)->Buffer)) ;
.catch { .block {  r $t0 = $spat( "${instr}", "*${$arg1}*" ) ; } } ;
.if (@$t0 !=1) {gc} .else {.echo matched ${instr} } ;

bps like below

kd> bl
 0 e fc34fc03     0001 (0001) Ntfs!NtfsReadMftRecord "$$>a< printdec.txt"
 1 e 8056cdc0     0001 (0001) nt!NtCreateFile "$$>a< filepattern.txt waxford"

and output on double clicking the waxford.txt file in vpc's Desktop till opening of the file in desktop

mft# =  10535	FileName = NTUSER.DAT.LOG
mft# =  10328	FileName = ADMIN
mft# =  10535	FileName = NTUSER.DAT.LOG
mft# =  10328	FileName = ADMIN
mft# =  10535	FileName = NTUSER.DAT.LOG
mft# =  10328	FileName = ADMIN
mft# =  10535	FileName = NTUSER.DAT.LOG
mft# =  10328	FileName = ADMIN
mft# =   2162	FileName = NOTEPAD.EXE
mft# =     87	FileName = APPPATCH
mft# =     29	FileName = SYSTEM32
mft# =      5	FileName = \
mft# =     28	FileName = WINDOWS
mft# =     29	FileName = SYSTEM32
mft# =      5	FileName = \
mft# =     28	FileName = WINDOWS
mft# =     29	FileName = SYSTEM32
mft# =      5	FileName = \
mft# =     28	FileName = WINDOWS
mft# =     29	FileName = SYSTEM32
mft# =     29	FileName = SYSTEM32
mft# =     29	FileName = SYSTEM32
mft# =     29	FileName = SYSTEM32
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =   3700	FileName = X86_POLICY.6.0.MICROSOFT.WINDOWS.COMMON-CONTROLS_6595B64144CCF1DF_X-WW_5DDAD775
mft# =     28	FileName = WINDOWS
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =  10535	FileName = NTUSER.DAT.LOG
mft# =  10328	FileName = ADMIN
mft# =  10535	FileName = NTUSER.DAT.LOG
mft# =  10328	FileName = ADMIN
mft# =  10328	FileName = ADMIN
mft# =  10535	FileName = NTUSER.DAT.LOG
mft# =  10328	FileName = ADMIN
Syntax error at '( "\DEVICE\HARDDISKVOLUME1\", "*waxford*" ) ; '
mft# =      5	FileName = \
Syntax error at '( "\DEVICE\HARDDISKVOLUME1\DOCUMENTS AND SETTINGS\", "*waxford*" ) ; '
mft# =   3646	FileName = DOCUME~1
mft# =   3646	FileName = DOCUME~1
mft# =  10328	FileName = ADMIN
mft# =  10471	FileName = DESKTOP
mft# =      5	FileName = \
mft# =   3646	FileName = DOCUME~1
mft# =  10328	FileName = ADMIN
Syntax error at '( "\DEVICE\HARDDISKVOLUME1\DOCUMENTS AND SETTINGS\ADMIN\", "*waxford*" ) ; '
mft# =  10455	FileName = LOCAL SETTINGS
mft# =  10328	FileName = ADMIN
mft# =  11678	FileName = WAXFORD.TXT.LNK
mft# =  11678	FileName = WAXFORD.TXT.LNK
Syntax error at '( "\DEVICE\HARDDISKVOLUME1\DOCUMENTS AND SETTINGS\ADMIN\DESKTOP\", "*waxford*" ) ; '
mft# =  11673	FileName = CHANGE.LOG
mft# =  11673	FileName = CHANGE.LOG
mft# =  11638	FileName = RP7
mft# =  11678	FileName = WAXFORD.TXT.LNK
mft# =  10451	FileName = RECENT
mft# =  10471	FileName = DESKTOP
mft# =      0	FileName = $MFT
mft# =  10535	FileName = NTUSER.DAT.LOG
mft# =  10328	FileName = ADMIN
Syntax error at '( "\DEVICE\HARDDISKVOLUME1\WINDOWS\", "*waxford*" ) ; '
mft# =  10535	FileName = NTUSER.DAT.LOG
mft# =  10328	FileName = ADMIN
mft# =     28	FileName = WINDOWS
mft# =  10451	FileName = RECENT
matched \??\C:\Documents and Settings\admin\Desktop\waxford.txt
nt!NtCreateFile:
8056cdc0 8bff            mov     edi,edi
kd> kb
ChildEBP RetAddr  Args to Child              
f91a7d30 804de7ec 00c4dbb0 00100080 00c4db50 nt!NtCreateFile
f91a7d30 7c90e4f4 00c4dbb0 00100080 00c4db50 nt!KiFastCallEntry+0xf8
00c4db0c 7c90d09c 7c8109a6 00c4dbb0 00100080 ntdll!KiFastSystemCallRet
00c4db10 7c8109a6 00c4dbb0 00100080 00c4db50 ntdll!ZwCreateFile+0xc
00c4dba8 7ca2560d 00c4e0f4 00000080 00000007 kernel32!CreateFileW+0x35f
00c4dc0c 7ca254e4 00c4e0f4 00c4dc2c 00c4dc30 SHELL32!GetFindDataForPath+0x26
00c4de84 7ca252ee 00c4e0f4 60000000 00ad0084 SHELL32!CShellLink::_GetFindDataAndTracker+0x4c
00c4e300 7ca2ed9d 00ad0084 00000000 00000001 SHELL32!CShellLink::_SetPIDLPath+0x15f
00c4e314 7ca3074a 00153fc8 00ad0084 00ad0084 SHELL32!CShellLink::SetIDList+0x14
00c4e758 7ca30590 00ad0084 00c4e790 00c4e78c SHELL32!CreateLinkToPidl+0x178
00c4e99c 7ca3023d 00ad0084 00ad006c 00c4e9c4 SHELL32!CTaskAddDoc::_CreateMRUItem+0x5b
00c4e9c8 7ca2ffc3 00ad0084 00ad006c 00ad007a SHELL32!CTaskAddDoc::_AddDocToRecentAndExtRecent+0x55
00c4fe4c 7ca2fece 00ad0084 00ad001c 0014f3a0 SHELL32!CTaskAddDoc::_AddToRecentDocs+0xf6
00c4fe68 7c9f47ed 00ad0084 00111660 000d71d8 SHELL32!CTaskAddDoc::RunInitRT+0x69
00c4fe84 75f81b9a 0014f398 75f81b18 75f80000 SHELL32!CRunnableTask::Run+0x54
WARNING: Frame IP not in any known module. Following frames may be wrong.
00c4fee0 77f69588 001612b0 00127e40 77f6956b 0x75f81b9a
00c4fef8 7c927aa2 00127e40 7c97b440 00127990 SHLWAPI!ExecuteWorkItem+0x1d
00c4ff40 7c927ae3 77f6956b 00127e40 0009cc50 ntdll!RtlpWorkerCallout+0x70
00c4ff60 7c927ba5 00000000 00127e40 00127990 ntdll!RtlpExecuteWorkerRequest+0x1a
00c4ff74 7c927b7c 7c927ac9 00000000 00127e40 ntdll!RtlpApcCallout+0x11
00c4ffb4 7c80b713 00000000 0007ccf4 0007ccf4 ntdll!RtlpWorkerThread+0x87
00c4ffec 00000000 7c910230 00000000 00000000 kernel32!BaseThreadStart+0x37
kd> g
mft# =  11672	FileName = WAXFORD.TXT
mft# =     25	FileName = $OBJID
Syntax error at '( "\DEVICE\HARDDISKVOLUME1\WINDOWS\APPPATCH\", "*waxford*" ) ; '
matched \DEVICE\HARDDISKVOLUME1\WINDOWS\APPPATCH\
nt!NtCreateFile:
8056cdc0 8bff            mov     edi,edi
kd> al
  Alias            Value  
 -------          ------- 
 instr            \DEVICE\HARDDISKVOLUME1\WINDOWS\APPPATCH\ 
 mftfilename      SOFTWARE.LOG 
kd> ad *
kd> al
No aliases
kd> g
mft# =     25	FileName = $OBJID
Syntax error at '( "\??\C:\Documents and Settings\admin\Desktop\", "*waxford*" ) ; '
mft# =  10471	FileName = DESKTOP
Syntax error at '( "\DEVICE\HARDDISKVOLUME1\WINDOWS\SYSTEM32\", "*waxford*" ) ; '
mft# =  10471	FileName = DESKTOP
mft# =  10451	FileName = RECENT
mft# =  10451	FileName = RECENT
mft# =     29	FileName = SYSTEM32
matched \??\C:\Documents and Settings\admin\Recent\waxford.txt.lnk
nt!NtCreateFile:
8056cdc0 8bff            mov     edi,edi
kd> g
mft# =  10451	FileName = RECENT
mft# =  10451	FileName = RECENT
Syntax error at '( "\DEVICE\HARDDISKVOLUME1\WINDOWS\WINSXS\", "*waxford*" ) ; '
matched \DEVICE\HARDDISKVOLUME1\WINDOWS\WINSXS\
nt!NtCreateFile:
8056cdc0 8bff            mov     edi,edi
kd> g
mft# =  10451	FileName = RECENT
mft# =      0	FileName = $MFT
Syntax error at '( "\DEVICE\HARDDISKVOLUME1\WINDOWS\WINSXS\", "*waxford*" ) ; '
matched \DEVICE\HARDDISKVOLUME1\WINDOWS\WINSXS\
nt!NtCreateFile:
8056cdc0 8bff            mov     edi,edi
kd> al
  Alias            Value  
 -------          ------- 
 instr            \DEVICE\HARDDISKVOLUME1\WINDOWS\WINSXS\ 
kd> g
mft# =  11673	FileName = CHANGE.LOG
mft# =  11673	FileName = CHANGE.LOG
mft# =  11638	FileName = RP7
mft# =  11678	FileName = WAXFORD.TXT.LNK
Syntax error at '( "\DEVICE\HARDDISKVOLUME1\WINDOWS\WINSXS\X86_MICROSOFT.WINDOWS.COMMON-CONTROLS_6595B64144CCF1DF_6.0.2600.5512_X-WW_35D4CE83\", "*waxford*" ) ; '
mft# =  11678	FileName = WAXFORD.TXT.LNK
mft# =  11678	FileName = WAXFORD.TXT.LNK
mft# =   3696	FileName = X86_MICROSOFT.WINDOWS.COMMON-CONTROLS_6595B64144CCF1DF_6.0.2600.5512_X-WW_35D4CE83
mft# =  10451	FileName = RECENT
mft# =  10535	FileName = NTUSER.DAT.LOG
mft# =   1944	FileName = SHELL32.DLL
mft# =  10328	FileName = ADMIN
mft# =  10455	FileName = LOCAL SETTINGS
mft# =     29	FileName = SYSTEM32
mft# =     87	FileName = APPPATCH
mft# =  11678	FileName = WAXFORD.TXT.LNK
mft# =     28	FileName = WINDOWS
mft# =     28	FileName = WINDOWS
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =     28	FileName = WINDOWS
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =   3700	FileName = X86_POLICY.6.0.MICROSOFT.WINDOWS.COMMON-CONTROLS_6595B64144CCF1DF_X-WW_5DDAD775
mft# =     28	FileName = WINDOWS
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =     29	FileName = SYSTEM32
mft# =     29	FileName = SYSTEM32
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =   3700	FileName = X86_POLICY.6.0.MICROSOFT.WINDOWS.COMMON-CONTROLS_6595B64144CCF1DF_X-WW_5DDAD775
mft# =     28	FileName = WINDOWS
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   4979	FileName = EN-US
mft# =   3665	FileName = POLICIES
mft# =     28	FileName = WINDOWS
mft# =    127	FileName = MANIFESTS
mft# =     28	FileName = WINDOWS
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =    180	FileName = EN
mft# =     29	FileName = SYSTEM32
mft# =   3630	FileName = SOFTWARE.LOG
mft# =   5507	FileName = <Win32 error 0n30>
mft# =   3630	FileName = SOFTWARE.LOG
mft# =   5507	FileName = <Win32 error 0n30>
mft# =  10471	FileName = DESKTOP
matched \??\C:\Documents and Settings\admin\Desktop\waxford.txt
nt!NtCreateFile:
8056cdc0 8bff            mov     edi,edi
kd> kb
ChildEBP RetAddr  Args to Child              
f8f2fd30 804de7ec 0007fe0c 80100080 0007fdac nt!NtCreateFile
f8f2fd30 7c90e4f4 0007fe0c 80100080 0007fdac nt!KiFastCallEntry+0xf8
0007fd68 7c90d09c 7c8109a6 0007fe0c 80100080 ntdll!KiFastSystemCallRet
0007fd6c 7c8109a6 0007fe0c 80100080 0007fdac ntdll!ZwCreateFile+0xc
0007fe04 01004a0d 0100a900 80000000 00000003 kernel32!CreateFileW+0x35f
0007fedc 01002980 01000000 00000000 00020630 svchost!___PchSym_+0x741
0007ff1c 01007511 01000000 00000000 000a2332 svchost!RegQueryStringA+0x86
WARNING: Frame IP not in any known module. Following frames may be wrong.
0007ffc0 7c817067 00150950 0007daf8 7ffd5000 0x1007511
0007fff0 00000000 0100739d 00000000 78746341 kernel32!BaseProcessStart+0x23
kd> bl
 0 e fc34fc03     0001 (0001) Ntfs!NtfsReadMftRecord "$$>a< printdec.txt"
 1 e 8056cdc0     0001 (0001) nt!NtCreateFile "$$>a< filepattern.txt waxford"

[Aimless]

Waxy...love ya!



Have Phun

[WaxfordSqueers]

yes you need two machines and a transport between them for windbg to work if you want to work with real hot iron

Appreciate all the work Blabs, hope I am not taking you away from something important.

In the old days, it required a serial cable between host and target. I don't know if USB -> USB has functionality for that these days. I have a LAN working between my laptop and desktop via NIC cards but it is unstable. Works for file transfer but it complains a lot and sometimes refuses to work.

I do have a VM setup (VMWare) with windows and I recall trying to start a pipe at one time between the VM as target and windbg on the host. For some reason it never got going. I'll play with it.

read the articles they both exist for a reason bloat with garter-belt is not one of them

http://msdn.microsoft.com/en-in/library/windows/hardware/ff565438%28v=vs.85%29.aspx
http://www.osronline.com/article.cfm?id=257

The second article is good and reflects what I read in a book by Russinovich (sysinternals and now microsoft guru). I'll have to read it more closely and poke around in the code. I understand the number passed in EAX is an index into the system service table and the pointer in EDX points to a user stack. Sysenter now makes perfect sense to me. However, if these funcs are identical as claimed by msoft in the first link, and we know how they tend to mislead users by withholding all the information, that does not explain why the softice driver can't deal with NtCreateFile (wont break on a bpx) yet works perfectly with ZwCreateFile.

My reference to bloatware was not to the Nt/Zw names and usage, it was to the whole idea of objects. Why not call a file a file and a disk drive a disk drive? What's the penchant for obfuscating real world objects with virtual creations that are so generalized they fail to make sense? Once again, I think the problem is with msoft trying to keep trade secrets too close to the vest.

I understand the need to virtualize certain facets of programming, to distinguish the real disks and files from their images in memory. But msoft is getting carried away developing a language and implementation that is so broad that someone could learn this jargon and never understand what a real disk drive or file is. It's like at university, where some profs teach math from an unrealistically broad set of definitions that first year students get totally messed up. I had a prof tell our class that one could take the square root of -1 in certain situation. I asked for an example and he just stuttered. I was a marked man from that moment on but I was not about to let some egghead spin bs about math. I know about the usage of the square root of -1 in complex numbers but no one takes the root of -1 there.

windbg knows ntfs better than me

started a vpc - kd session

asked what windbg knows about ntfs,mft,read,write with

found this part really interesting but I need to get up to speed on what you are talking about. The ref to Ntfs!NtfsReadMftRecord is interesting. I'll have to sift through ntoskrnl symbols and see if there are other NFT references. Maybe even the HAL symbols.

What I am trying to do is approach the $MFT file from its entry point and trace through it's front end so I can see where it goes, especially from the $ROOT file.

Each of the first set of files are metafiles and begin with the header 'FILE'. The very first entry in an MFT file is FILE, like MZ is the first entry in an exe file. From there onward, there are attributes at various offsets, and each attribute begins with a number. It's like a PE header and I want to learn to read it.

Trouble is, as I understand it, once the disk has been accessed, NTFS reads the files and moves info to another location. Also, a file like Notepad, if used regularly, will have a prefetch record and the MFT will likely be bypassed. So I need to catch the NTFS system in the act of initializing the MFT. I want to see what it's initial entry point is into the table, and why.

[WaxfordSqueers]

Waxy...love ya!



Have Phun

Worked with a guy years ago and he always posed the question, "Having fun yet"?

[blabberer]

the free vmware player is more than sufficient and you can use vmkd a superfast transport in this
the free vpc is quiet good and the virtual serial interface with namedpipe on com1 is what i normally use
the free virtualbox has its own followers and iirc source code is available if you are bent on recompiling it with some super duper crap

all of these work very very superbly for kernel debugging in virtual environment without having a second machine / and / or comport (a rarity nowadays)/ null modem cable / firewire / 1394 / debugusb / network (yes real tcp network transport is available in win8 the touch touch feel feel duplo crap ui)

set aside three to four hours and decide to make it work and you wont regret dumping softice and its quirks and limitations.

square root of -1 = i or -i

ntfs!ntfsReadMftRecord is a ntfs symbol you need to look int ntfs.pdb for it not in ntoskrnl symbols

if you were on windbg it would be as simple as .reload /f ntfs.sys and then using x (examine )
in softice you would need the .nms or some such file for ntfs.sys

i am not sure how reading memory with debuggers is going to get you anything from a physical media
my understanding till now regarding recovery is you need some disk utility to read the physical media scanning for patterns
and getting deep from there so since it is offline / static / //// imur nothing is going to move from anywhere to anywhere in a physical disk
or do they move even if you do read only access ?? iam absolutely no good at this game ( i make backups and simply zero out dod style fdisk and repartition and reformat and install new

[WaxfordSqueers]

square root of -1 = i or -i

Appreciate the info on vmware, my info is outdated. Forgot that ntfs.sys is a file. Better check it out. It's no problem making an nms for softice but I will look into windbg.

BTW, the derivation of i is i^2 + 1 = 0. therefore i = root(-1). The i means nothing other than in applications like electrical engineering but the root of -1 is never taken. i forms a complex plane with real numbers in which i is imaginary. That imaginary component can be used to represent reactive power in an electrical motor, which represents the reactive power required to maintain the motor's electric fields, for it's magnets. The real component represents real power.

It becomes a big issue with Hydro as more and more magnetic devices come on line from factories. The current through an inductively reactive device is out of phase with the applied voltage, and as more reactive devices come on line, the phase of the current lags the voltage more and more. It's called the power factor and a unity power factor is ideal, meaning the current is in phase with the voltage, which is the case in a purely resistive device. As it lags, the pf drops down below 90%, and Hydro gets worried because it's supplying a lot of power to feed the magnetic fields in reactive devices that is doing no good work.

In factories, they have huge banks of capacitors, through which current leads voltage. By applying the capacitor banks across a motor load, it brings the lagging current closer to unity by offsetting the inductive load.

i am not sure how reading memory with debuggers is going to get you anything from a physical media
my understanding till now regarding recovery is you need some disk utility to read the physical media scanning for patterns....

When you call a file, the windows kernel has to i/f with the physical media to retrieve the data. Since the MFT in NTFS is regarded as one large file, something has to read it. I am not trying to recover data, I am trying to repair a link that has been broken in the MFT. Most of my data is sitting there in files and directories but the link is broken. In essence, I am troubleshooting (reversing) a broken link in the MFT.

MY MO is to find how Windoze reads the disk, especially the MFT, so that I can look up in the real MFT how to access directories and files. I plan to do that by reading a known good NTFS MFT table. If I can repair the broken link, I may be able to put the files system back the way it was before I messed up.

I would imagine HAL, or something, reads the standard 512 bytes from disk, or multiples thereof, and reads it into a buffer. Then something must process the buffer to get the info it needs to access files stored in a binary tree. Otherwise, how would they ever retrieve a file of any kind, or know where to read a directory? The system must know where the MFT pointer is located in the boot record section and it must know where to go in the MFT to get the file attributes.

I have found nothing on the Net that goes into it at such a low-level.

I have a couple of excellent disk editors, both freeware. One of them, Active@ Disk Editor, allows me to write to disk much like Norton Disk Editor. It also has a really good facility (called a template) that allows me to place the cursor on the 'FILE' header of an MFT file, and it will tell me, using tooltips, what each attribute means. It even colours each attribute a different colour. Another template, when placed on the boot sector, tells you what each offset means.

The other disk program (testdisk), recovers files by reading the file signature. So far, it has recovered most of the known files on the disk but I am look for one that was a backup of another computer. I don't think testdisk would have the signature for that file since it was a Microsoft backup. I have since made a good backup of that computer. It does not always recover the original filename, so I have to open each file to see what it is. With exes that can be dangerous, as you know, so I look up the file properties or use a hex editor to read the file info in the resource section.

NTFS uses a numbering system for files that I need to learn how to decode. Small files, under about 1K are kept in the MFT and called 'resident'. However, larger files have only their front end in the MFT and reside outside the MFT and are called non-resident. Each non-resident portion is pointed to by the file number, which lists the offset and length. As it stands, I don't even know where to look for the file number. It's in an attribute but I don't have a template for the $ROOT file, which is the root directory in NTFS. I thought I might be able to trace into it through Ntoskrnl and HAL, or some other low-level access.

I pulled a bonehead move and used Comodo Backup in clone mode thinking I was creating a disk image. It ran for several seconds before I realized that and I managed to stop it. The mirror MFT is intact and I can replace the damaged MFT portion covered by it but I am not sure yet if too much as been damaged. I wont know until I trace into the MFT to see where the break is located.

I don't know if Comodo did a bit by bit clone, beginning at the the first sector, or whether it has an algorithm whereby it writes the MFT first. The MFT is well into the disk(cluster 0xC0000) and I am hoping the clone program did not alter it too much. It obviously did something but I am not sure as to the extent of the damage yet. Many of the original MFT files are still there...I think.

It's not critical that I recover the data. I am just curious as to whether i can reverse the damage, and it gives me a chance to delve into NTFS and the kernel.

[blabberer]

yep i understand real numbers and complex numbers and power factor have tinkered with an automatic power factor corrector that connects and disconnects capacitors as needed for radyne / elind induction heating furnaces / esab welding transformers

like i said i was wondering what this ntfs is about and opened my harddisk in a simple fopen kinda program
and tried looking for FILE0 and FileName

it seems decoding this is not much of a deal
google is strife with info and all it would take is a bit of dedicated hours

my simple program using pure crt lib functions like fopen , fread , ftell , & printf
can open my harddisk of laptop (about 112 gb) and is able to list almost all of the first 2000 files names i asked from it
in reasonable minutes

#include "stdafx.h"
#include <windows.h>
//#define printdump

// unfinished ntfs mft structure attribute data needs to be filled in

typedef struct _NTFSMFT {
    DWORD MAGIC;
    WORD UpdateSeqOffset;
    WORD FixupArrayEntries;
    DWORD64 $LogFileSeqNo;
    WORD SequenceNumber;
    WORD HardLinkCount;
    WORD AttributeOffset;
    WORD Flags;
    DWORD MftUsed;
    DWORD MftAlloc;
    DWORD64 FileRefernace;
    WORD NextAttributeID;
    WORD AlignNext4B;
    DWORD ThisMFTRecordNumber;
    BYTE FixupData[0xD0];
} NtfsMft, *PNtfsMft;

int _tmain(int argc, _TCHAR* argv[])
{
    printf("lets open the harddisk and look at it \n");
    FILE * fp;
    BYTE dump[0x260];
    memset(&dump,0,sizeof(dump));
    fopen_s( &fp,"\\\\.\\PHYSICALDRIVE0", "rb" );
    int foo = -1;
    do

    {
        foo++;
        do
        {
            fread_s(dump,0x260,1,0x200,fp);  // read one sector
            int i = -1;
            do 
            {
                do 
                {
                    i++;
                    // first do while looks recursively for 'F' max one sector
                    // second do while looks for "FILE0" max one sector
                    // third do while traverses sectors till 2000 'FILE0' are not found
                }while ((dump[i] != 0x46) && (i<=0x1ff) ); 
            } while ( ( (*(DWORD64 *)dump & 0xffffffffff) != 0x30454c4946) && (i<0x1ff)  );
        }while      ( ( (*(DWORD64 *)dump)& 0xffffffffff) != 0x30454c4946) ;
        BYTE temp1 = dump[0x14]; // see struct above Attribute Offset
        BYTE temp2 = dump[temp1+0x4]; // StdInfo Attribute Length
        BYTE temp3 = dump[temp1+temp2+0x10]; // name Info wstr Start nneds more work to be good 30% miss rate 
        DWORD temp4 = temp1+temp2+temp3; // needed for FileName wstr address
        USHORT NameLen = dump[temp4-2]; // name info wstr max len needed for printf  see printf below
        printf ("FILE0 found at %I64x file name is %.*S\n",(_ftelli64(fp)-0x200),NameLen,&dump[temp4]);
#ifdef printdump  // debug aid will dump 200 bytes from matched postion uncomment the #define
        int k = 0;
        while (k <0x200)
        {
            for(int i = k; i<k+0x10; i++)
            {
                printf("%02x ",(BYTE) dump[i]);
            }
            for (int i = k; i < k+0x10; i++)
            {
                printf ("%c ",(BYTE) dump[i]);
            }
            printf("\n");
            k+=0x10;
        }
#endif printdump
    } while(foo < 0x2000); // exit after finding 2000 signatures
    return TRUE;
}

result as follows

lets open the harddisk and look at it 
FILE0 found at 7a600 file name is CA0834~1.CAT
FILE0 found at 7aa00 file name is CAFAEA~1.CAT
FILE0 found at a19200 file name is DEVTES~1.HXI
FILE0 found at a19600 file name is DEVTES~1.HXS
FILE0 found at a19a00 file name is 
FILE0 found at a5c600 file name is 
FILE0 found at a5ca00 file name is MF_140~1.IGS
FILE0 found at a5ea00 file name is UKADIC~1.DOC

cutt

FILE0 found at 2bdeede00 file name is WINPCA~1.URL
FILE0 found at 2bdeee200 file name is 
FILE0 found at 2bdeee600 file name is 
FILE0 found at 2bdeeea00 file name is CO3267~1.XML
FILE0 found at 2bdeeee00 file name is LM0201~1.LOG

[WaxfordSqueers]

it seems decoding this is not much of a deal
google is strife with info and all it would take is a bit of dedicated hours

ooooooh...some people's kids. All I have been doing the past week is scouring the net and reading book after book on the NTFS and the Windows kernel. I have forgotten what sunlight looks like.

I appreciate the work you have done on this but your program is far too simple. I can find all the FILE headers using Active@ and I can see files in there as well, but it's not as straight forward finding a file in the MFT and putting a directory back together. The FILE header has nothing to do with the files per se, it is a header for metafiles, which describe the system. There are at least 30 different metafiles, all beginning with the FILE header.

eg. hope the formatting holds

Inode------Filename-----------Description

0-----------$MFT----------Master File Table - An index of every file...pointer to $MFT start cluster in Master Boot Record.
1-----------$MFTMirr------A backup copy of the first 4 records of the MFT....ptr to $MftMirr start cluster in MBR.
2-----------$LogFile------Transactional logging file...chkdsk writes to this log and uses it to roll back errors during caching.
3-----------$Volume-------Serial number, creation time, dirty flag....you can see disk volume name here....dirty flag for chkdsk.
4-----------$AttrDef------Attribute definitions
5-----------. (dot)-------Root directory of the disk
6-----------$Bitmap-------Contains volume's cluster map (in-use vs. free)
7-----------$Boot---------Boot record of the volume
8-----------$BadClus------Lists bad clusters on the volume
9-----------$Quota---------Quota information
10----------$UpCase-------Table of uppercase characters used for collating

These are the first 10 files (records) in the MFT and they all begin with 'FILE'. To make matters confusing, everything is a file and each file has attributes. File #6 at inode 5 is called the dot file as in '.' of normal directory roots. That's where you need to look for directories and files, but as you might guess, directories are files as well, and they have attributes.

The top node of the B-tree structure used to store files and directories is located in the dot file.

Each one of those files has attributes as follows:

Type----------Name----------
0x10----------STANDARD_INFORMATION-----file creation date, etc.
0x20----------$ATTRIBUTE_LIST----------used when there lots of files that wont fit into the MFT
0x30----------$FILE_NAME---------------stores name of file attribute
0x40----------$VOLUME_VERSION----------
0x50----------$SECURITY_DESCRIPTOR
0x60----------$VOLUME_NAME-------------name of volume
0x70----------$VOLUME_INFORMATION
0x80----------$DATA--------------------contains information about files data i.e. the actual files.
0x90----------$INDEX_ROOT--------------root node of the data file/directory B-tree
0xA0----------$INDEX_ALLOCATION
0xB0----------$BITMAP-------------has similarities to the FAT, but very loosely
0xC0----------$SYMBOLIC_LINK
0xD0----------$EA_INFORMATION
0xE0----------$EA
0xF0----------$PROPERTY_SET


So, the first table is the actual metafiles and items from the 2nd table are
found in whichever metafile needs them for description.

For example, the STANDARD_DEFINTION begins with an 0X10. The data (actual files) section begins with 0x80.

If you want to see how they are laid out, download the free app Active@ Disk Editor and have a look at the templates section. You can find the address of the MFT by using the template for the master boot record. Then look up the MFT and it will begin with 'FILE'. Align the template over the beginning of FILE and you'll get a nice colour coded layout with tooltips that reveal what each part means as you hover the mouse over them.

I understand all of that quite well now, after hours and hours of reading,
what I am looking for now is how to interpret entries in the atrributes that
describe how files are connected from the the root out.

I thought one way would be to trace the system using softice because I
have not seen an adequate description of how that is done. In other words,
I am breaking new ground, as far as I know. Someone has no doubt done it
and I have seen a program that is trying to do it. The code is here:

http://www.autoitscript.com/forum/topic/94269-mft-access-reading-parsing-the-master-file-table-on-ntfs-filesystems/

Here's the source of my two tables above:

http://dubeyko.com/development/FileSystems/NTFS/ntfsdoc.pdf

and another http://grayscale-research.org/new/pdfs/NTFS%20forensics.pdf

[WaxfordSqueers]

my simple program using pure crt lib functions like fopen , fread , ftell , & printf
can open my harddisk of laptop (about 112 gb) and is able to list almost all of the first 2000 files names i asked from it
in reasonable minutes

If it interests you, I have found an interesting app at the Code Project, with source code for helper library headers and samples of the code, that claims to parse the MFT.

http://www.codeproject.com/Articles/81456/An-NTFS-Parser-Lib

At this link, there is code for an undelete app that uses the headers from above:

http://read.pudn.com/downloads149/sourcecode/windows/system/642135/Undelete/NTFSDrive.cpp__.htm

I saw in another thread that there is a problem with the compiled version of this code, but it seems to be minor. I am trying to find the reference again and I'll forward it if I do.

[blabberer]

ooohh some peoples fathers ........ glad i provoked and glad you replied this thread atleast gathers moss

the autoit script guy has a google code page that has a gew more autoit scripts for extracting files as well apart from the mft2csv

i just read the greyscale pdf (couldnt download when i was searching) has code similar to what i wrote and the unfinished structure is completely same
apart from the fact that i have a 0xd0 filling that is AttrData left out pending research

apart from them alex ionescu seems to have decoded ntfs in visual basic when he was probably running in diapers (some peoples children ....)

ntfsdoc is the mother of all docs it seems by russon & fledel (referenced from lot of pages that has the keyword NTFS)

thomas schwarz ntfsfs forensics ppt mft structure is what i used when i cooked the code above

like i said ntfswalker by dmitrybyrant is a free program that displays all these tables in a bit understandable format
right from sequence id 0 to N

btw plain hxd hexeditor does a fine job of showing me the raw layout of hdd which i collated with ntfs walker and the output of my program

i accept it is not as trivial as i made it to be but if you persist and i persist we may be able to document half the ntfs in this thread

as both of us has got nothing to lose or no money to gain but just pure satisfaction of doing something we like and doing it in the best possible way

have fun

hope you finally unravel how to traverse the ntfs web like a spider getting to anywhere from anywhere and everywhere methodically


by methodically i mean we can probably extract and then use secure erase / wipe to zero the extracted file and the harddisk becomes clutter free

and with the process of extraction and elimination we have less and less data to decode
and one day some day the hard disk will be clean without any files just as plain as a newly fdisked formatted factory sealed disk

[WaxfordSqueers]

hope you finally unravel how to traverse the ntfs web like a spider getting to anywhere from anywhere and everywhere methodically

I'll answer your post in more detail later. It's 5 am and I am getting burnt. I am supposed to be out exercising, not getting myself bleary eyed traversing the ntfs web. However, it is intriguing.

Made some headway tonight with good old softice and some basic hacking. I have no idea at this point what kind of breakpoint to set or which one. So, I tried SetFilePointer with ReadFile. Then I set a a BMSG on LBUTTONDBLCLK (don't use that often) with a HWND I found in SPYXX for the file manager I use. SPYXX is kinda neat because you can set the window to spew messages for a particular HWND and it tells you which window spewed them. Only one was spewing the double click message, so I did a BMSG on that one with the handle and softice popped up in my file manager after I double-clicked notepad.

From there I tried F5 (go in softice) with both SetFilePointer and ReadFile set, but I was getting way too many hits on ZwReadFile. So I did it with just SetFilePointer and F5'd till notepad popped up. It took 16 hits, so I reset and broke after 14 hits, then traced. It was not long till I hit NTFS.SYS, which seemed encouraging.

Somewhere along the way I hit FsRtlCopyRead in Ntoskrnl at offset B7E5CE94. The literature says that function does a fast copy read from a cached file to a buffer. A bit later, in the function call, there is a call to _CcGetActiveVACB, where VACB is the virtual address control block. The lit claims that if a file mapped into the VACB has a name, the name gets listed. However, if it has no name, the VACB is caching metadata. Metadata is the magic word in my mind for MFT stuff. Since I just came through NTFS.SYS, it seemed reasonable that ref to metadata meant the MFT.

They showed an example related to the VACB using the kernel debugger as follows:

kd> !filecache

It returned a list with many MFT records in it.

It's not clear to me at the moment the difference between windbg and kd. Does kd require the same rigmarole as windbg or is it just part of the same package? Or is kd the command line part and windbg the GUI?

It seems that windows reads the MFT records while accessing the drive initially and caches them for future use. That's where I left off tonight. I am hoping that if I dig further into this that I will be able to see how Win and NTFS access the metadata to find a file. I have turned off prefetch but I am not sure if Windoze caches files in the same way it does prefetch data. If it does, I need to find a way to flush it's cache so it starts over. Or, approach it using my other scheme of unplugging the external drive, rebooting the OS, and plugging the drive back in with a breakpoint on an I/O port. Have not tried that before so it would be new ground for me.

In Win 7 you can set the drive offline using the MMC. I don't think there's a way to do that in XP.

Have fun yourself, or phun, as aimless poots it.