Archive for the ‘Notes on Advanced .NET Debugging’ Category

Reading Notebook: 25-January-10

Monday, January 25th, 2010

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

Kernel Process variables (p. 343)

0: kd> !process poi(PsIdleProcess)
PROCESS fffff800019910c0
    SessionId: none  Cid: 0000    Peb: 00000000  ParentCid: 0000
    DirBase: 00124000  ObjectTable: fffff88000000080  HandleCount: 606.
    Image: Idle
    VadRoot fffffa8003b97c70 Vads 1 Clone 0 Private 1. Modified 0. Locked 0.
    DeviceMap 0000000000000000
    Token                             fffff88000003330
    ElapsedTime                       00:00:00.000
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         0
    QuotaPoolUsage[NonPagedPool]      0
    Working Set Sizes (now,min,max)  (6, 50, 450) (24KB, 200KB, 1800KB)
    PeakWorkingSetSize                6
    VirtualSize                       0 Mb
    PeakVirtualSize                   0 Mb
    PageFaultCount                    1
    MemoryPriority                    BACKGROUND
    BasePriority                      0
    CommitCharge                      0

        THREAD fffff80001990b80  Cid 0000.0000  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 0
        Not impersonating
        DeviceMap                 fffff88000007310
        Owning Process            fffff800019910c0       Image:         Idle
        Attached Process          fffffa8003bf1040       Image:         System
        Wait Start TickCount      16021          Ticks: 13224 (0:00:03:26.295)
        Context Switch Count      142852            
        UserTime                  00:00:00.000
        KernelTime                00:06:13.700
        Win32 Start Address nt!KiIdleLoop (0xfffff80001876880)
        Stack Init fffff80002bdadb0 Current fffff80002bdad40
        Base fffff80002bdb000 Limit fffff80002bd5000 Call 0
        Priority 16 BasePriority 0 PriorityDecrement 0 IoPriority 0 PagePriority 0
        Child-SP          RetAddr           Call Site
        fffff800`02bdad80 fffff800`01a43860 nt!KiIdleLoop+0x11b
        fffff800`02bdadb0 00000000`00000000 nt!zzz_AsmCodeRange_End+0x4

        THREAD fffffa60005f5d40  Cid 0000.0000  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 1
        Not impersonating
        DeviceMap                 fffff88000007310
        Owning Process            fffff800019910c0       Image:         Idle
        Attached Process          fffffa8003bf1040       Image:         System
        Wait Start TickCount      0              Ticks: 29245 (0:00:07:36.224)
        Context Switch Count      162365            
        UserTime                  00:00:00.000
        KernelTime                00:06:14.808
        Win32 Start Address nt!KiIdleLoop (0xfffff80001876880)
        Stack Init fffffa600191bdb0 Current fffffa600191bd40
        Base fffffa600191c000 Limit fffffa6001916000 Call 0
        Priority 16 BasePriority 0 PriorityDecrement 0 IoPriority 0 PagePriority 0
        Child-SP          RetAddr           Call Site
        fffffa60`0191bd80 fffff800`01a43860 nt!KiIdleLoop+0x11b
        fffffa60`0191bdb0 00000000`00000000 nt!zzz_AsmCodeRange_End+0x4

Relevant process functions (pp. 344 - 345) - More of them can be found here: http://msdn.microsoft.com/en-us/library/ms684847(VS.85).aspx

Protected processes (pp. 346 - 348) - It can be seen in _EPROCESS block (the output taken from a complete memory dump):

0: kd> dt _EPROCESS fffffa8004b5e040
ntdll!_EPROCESS
[...]
   +0x36c ProtectedProcess : 0y1
[...]

The following script lists protected processes on W2K8:

0: kd> !for_each_process "dt _EPROCESS ImageFileName @#Process; dt _EPROCESS ProtectedProcess @#Process"
ntdll!_EPROCESS
   +0x238 ImageFileName : [16]  "System"
ntdll!_EPROCESS
   +0x36c ProtectedProcess : 0y1
[...]
ntdll!_EPROCESS
   +0x238 ImageFileName : [16]  "audiodg.exe"
ntdll!_EPROCESS
   +0x36c ProtectedProcess : 0y1
[...]

System process is protected because of Ksecdd.sys stores info in user space (p. 347)

PROCESS_QUERY_LIMITED_INFORMATION (p. 347)

Access violation by design for Protected Media Path processes when a kernel-mode debugger is enabled (p. 348) - this is not an optimal design in my opinion - I had problems with that: http://www.dumpanalysis.org/blog/index.php/2010/01/08/live-kernel-debugging-of-a-system-freeze-case-study/. The better way is to show a message box and gracefully exit and only emit AV if message box is bypassed. 

Advanced .NET Debugging by M. Hewardt:

PE format and its relation to .NET (pp. 26 - 27)

AddressOfEntryPoint (pp. 28 - 29 and p. 31) - we can also use !dh command to find that address (similar to what dumpbin.exe does):

0:001> lm m notepad
start             end                 module name
00000000`ff180000 00000000`ff1af000   notepad    (deferred)        

0:001> !dh 00000000`ff180000
[...]
OPTIONAL HEADER VALUES
     20B magic #
    8.00 linker version
    E400 size of code
   1CC00 size of initialized data
       0 size of uninitialized data
    D1B4 address of entry point
    1000 base of code
         —– new —–
00000000ff180000 image base
    1000 section alignment
     200 file alignment
       2 subsystem (Windows GUI)
    6.00 operating system version
    6.00 image version
    6.00 subsystem version
   2F000 size of image
     400 size of headers
   32C26 checksum
[…]

0:001> u 00000000`ff180000+D1B4
notepad!WinMainCRTStartup:
00000000`ff18d1b4 4883ec28        sub     rsp,28h
00000000`ff18d1b8 e88b020000      call    notepad!_security_init_cookie (00000000`ff18d448)
00000000`ff18d1bd 4883c428        add     rsp,28h
00000000`ff18d1c1 e9b6fcffff      jmp     notepad!IsTextUTF8+0xc0 (00000000`ff18ce7c)
00000000`ff18d1c6 cc              int     3
00000000`ff18d1c7 cc              int     3
00000000`ff18d1c8 cc              int     3
00000000`ff18d1c9 cc              int     3

Application domains in ASP.NET; 3 default app domains (system, shared, default) in normal app (p. 34)

!dumpdomain SOS command (pp. 35 - 36)

Low(High)FrequencyHeap and StubHeap (p. 36) - Looks like they are not normal heaps or heap segments. I plan to test all commands on x64 .NET:

0:003> !dumpdomain
--------------------------------------
System Domain: 000007fef15a8ef0
LowFrequencyHeap: 000007fef15a8f38
HighFrequencyHeap: 000007fef15a8fc8
StubHeap: 000007fef15a9058
Stage: OPEN
Name: None
--------------------------------------
Shared Domain: 000007fef15a9860
LowFrequencyHeap: 000007fef15a98a8
HighFrequencyHeap: 000007fef15a9938
StubHeap: 000007fef15a99c8
Stage: OPEN
Name: None
Assembly: 0000000000372d10
--------------------------------------
Domain 1: 0000000000360840
LowFrequencyHeap: 0000000000360888
HighFrequencyHeap: 0000000000360918
StubHeap: 00000000003609a8
Stage: OPEN
SecurityDescriptor: 00000000003630e0
Name: TestCLR.exe
[...]

- Dmitry Vostokov @ SoftwareGeneralist.com

Reading Notebook: 07-December-09

Tuesday, December 8th, 2009

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

WMI CIM Studio (pp. 321 - 322)

dynamic and static MOF classes (p. 323) 

WbemTest, BMF (binary MOF), Mofcomp.exe (p. 323)

Object keys as WMI class instance specifiers (\\computer\root\namespace:class_name.Key1=”…”, Key2=”…”, … ) (pp. 324 - 325)

WMI association classes (p. 325) 

WQL exampe (p. 327)

wmiprvse.exe as a WMI provider host (p. 327)

wmic.exe (p. 328)

Namespace level WMI secutiry (p. 329)

WDI, Windows Diagnostic Infrastructure and its instrumentation, DiagLog, SEM Scenario Event Mapper, on-demand diagnosis (pp. 329 - 330) - looks interesting, especially in the context of possible first fault software problem solving techniques (OpenTask has published a book on this topic: http://www.dumpanalysis.com/First+Fault+Software+Problem+Solving)

Advanced Windows Debugging by M. Hewardt and D. Pravat:

LRPC_CCALL(ADDRESS) vs. OSF_CCALL(ADDRESS) vs. DG_CCALL(ADDRESS) (pp. 389 - 390)

Undocumented MSRPC (p. 391) - there is an empirical technique to find LRPC server endpoint: http://www.dumpanalysis.org/blog/index.php/2008/07/11/in-search-of-lost-pid/

!lpc message (p. 393) - some additional scenarios can be found in patterns: http://www.dumpanalysis.org/blog/index.php/2008/12/17/crash-dump-analysis-patterns-part-42e/, http://www.dumpanalysis.org/blog/index.php/2007/11/29/crash-dump-analysis-patterns-part-9d/ and various case studies involving LPC chains: http://www.dumpanalysis.org/blog/index.php/pattern-cooperation/

_PS_IMPERSONATION_INFORMATION (p. 395) - Looks like on W2K8 x64 it is another bit union:

lkd> dt -r _ETHREAD
[…]
  +0×3b0 ClientSecurity   : _PS_CLIENT_SECURITY_CONTEXT
      +0×000 ImpersonationData : Uint8B
      +0×000 ImpersonationToken : Ptr64 Void
      +0×000 ImpersonationLevel : Pos 0, 2 Bits
      +0×000 EffectiveOnly    : Pos 2, 1 Bit

RPC cell debugging configuration (pp. 397 - 398)

Advanced .NET Debugging by M. Hewardt:

Lutz Roeder’s .NET Reflector (pp. 15 - 16)

Roberto Farah’s PowerDbg (pp. 17 -18)

MDA Managed Debugging Assistants (pp. 19 - 21) - looks similar to WDI (Windows Diagnostic Infrastructure) on-demand diagnostics for unmanaged code mentioned in Windows Internals book

CLI(+BCL) -> CLR (p. 24)

Rotor (p. 25) - looks like it has the same value as WINE for unmanaged code: http://www.dumpanalysis.org/blog/index.php/2006/11/16/how-wine-can-help-in-crash-dump-analysis/ 

- Dmitry Vostokov @ SoftwareGeneralist.com -

Reading Notebook: 25-November-09

Wednesday, November 25th, 2009

Comments in italics are mine and express my own views, thoughts and opinions

Windows Internals by M. Russinovich, D. Solomon and A. Ionescu:

FailureActionsOnNonCrashFailures (p. 310)

WaitToKillApp(Service)Timeout (p. 311)

Shutdown ordering and preshutdown notification (pp. 312 - 313)

Shared services vulnerability to a crashing bug (p. 313) - Because an exception in one thread doesn’t affect another thread if there is no dependency (see MTCrash application, http://www.dumpanalysis.org/blog/index.php/2008/12/31/mtcrash/) if we preserve the crashed process, for example, using Crash2Hang tool (http://www.dumpanalysis.org/blog/index.php/2008/12/29/crash2hang/) we might temporarily preserve functionality of the remaining services (if there is no dependency)

CNG-KeyIso service (p. 313)

Viewing services inside processes (pp. 315 - 316) - We can also see them in Task Manager when we sort Processes by PID:

SubProcessTag (p. 316) - Here is an example from svchost.exe PID 1016 from the screenshot above:

lkd> !process 0n1016 1f
Searching for Process with Cid == 3f8
Cid Handle table at fffff88008156000 with 1063 Entries in use
PROCESS fffffa8004adec10
    SessionId: 0  Cid: 03f8    Peb: 7fffffdd000  ParentCid: 0280
    DirBase: add75000  ObjectTable: fffff88007f3c4d0  HandleCount: 436.
    Image: svchost.exe
    VadRoot fffffa80048b9220 Vads 153 Clone 0 Private 1630. Modified 1512. Locked 6.
    DeviceMap fffff8800802ef40
    Token                             fffff880080aa060
    ElapsedTime                       5 Days 01:31:56.632
    UserTime                          00:00:05.257
    KernelTime                        00:00:04.555
    QuotaPoolUsage[PagedPool]         132496
    QuotaPoolUsage[NonPagedPool]      21488
    Working Set Sizes (now,min,max)  (3650, 50, 345) (14600KB, 200KB, 1380KB)
    PeakWorkingSetSize                3725
    VirtualSize                       78 Mb
    PeakVirtualSize                   84 Mb
    PageFaultCount                    38144
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      3976

[...]

        THREAD fffffa8004b55060  Cid 03f8.046c  Teb: 000007fffff9e000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Alertable
            fffffa8004b54a80  NotificationEvent
            fffffa8004b52a50  SynchronizationEvent
            fffffa8004b55e00  NotificationEvent
            fffffa8004b55118  NotificationTimer
        Not impersonating
        DeviceMap                 fffff8800802ef40
        Owning Process            fffffa8004adec10       Image:         svchost.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      28044441       Ticks: 4968 (0:00:01:17.501)
        Context Switch Count      3784            
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address dhcpcsvc6!Dhcpv6Main (0×000007fefd726884)
        Stack Init fffffa6003c47db0 Current fffffa6003c47230
        Base fffffa6003c48000 Limit fffffa6003c42000 Call 0
        Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           Call Site
        fffffa60`03c47270 fffff800`018a46fa nt!KiSwapContext+0×7f
        fffffa60`03c473b0 fffff800`018a9feb nt!KiSwapThread+0×13a
        fffffa60`03c47420 fffff800`01b03a8e nt!KeWaitForMultipleObjects+0×2eb
        fffffa60`03c474a0 fffff800`01b040d3 nt!ObpWaitForMultipleObjects+0×26e
        fffffa60`03c47960 fffff800`018a1ef3 nt!NtWaitForMultipleObjects+0xe2
        fffffa60`03c47bb0 00000000`776e72ca nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffffa60`03c47c20)
        00000000`0272f5e8 00000000`7758bc03 ntdll!ZwWaitForMultipleObjects+0xa
        00000000`0272f5f0 000007fe`fd726117 kernel32!WaitForMultipleObjectsEx+0×10b
        00000000`0272f700 000007fe`fd726944 dhcpcsvc6!ProcessDhcpv6RequestForever+0×143
        00000000`0272f7c0 00000000`7758be3d dhcpcsvc6!Dhcpv6Main+0xc0
        00000000`0272f800 00000000`776c6a51 kernel32!BaseThreadInitThunk+0xd
        00000000`0272f830 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

[...]

lkd> dt _TEB 000007fffff9e000 SubProcessTag
ntdll!_TEB
   +0x1720 SubProcessTag : 0x00000000`00000011

Advanced .NET Debugging by M. Hewardt:

Debugging Tools for Windows (pp. 3 -4) - Here are quick links for download: http://windbg.org

No major CLR changes for .NET 3.x (p. 5)

DbgClr (p. 6)

MSBUILD XML example (pp. 6 - 7)

.load vs. .loadby (pp. 8 - 11) - Some additional load scenarios for legacy SOS and its server version can be found in comments to Managed Code Exception pattern: http://www.dumpanalysis.org/blog/index.php/2007/07/20/crash-dump-analysis-patterns-part-17/

SOSEX (pp. 10 - 11) - Added to my blog roll and links on http://DumpAnalysis.org

CLR Profiler (pp. 11 - 13) - Looks similar to functionality of unmanaged UMDH tool (user mode heap stack trace database)

- Dmitry Vostokov @ SoftwareGeneralist.com -