|
|
|
|
|
|
|
Windows内核调试
帖子发起人: Da Vinci 发起时间: 2009-08-06 13:02 下午 回复: 9
|
帖子排序:
|
|
|
|
2009-08-06, 13:02 下午
|
Da Vinci
注册: 2008-11-03
发 贴: 38
|
|
|
上周系统hang住的问题,现在又让对方同事拿到一个dump,初步分析一下,还是不太清楚是否是死锁的问题,想请张老师和各位老师帮忙看一下。现象就是系统logon之后,explorer显示之后会hang住,ctrl+del+alt不能调出task manager,鼠标不能用,不清楚是不是deadlock了。手动触发了一个蓝屏(http://msdn.microsoft.com/en-us/library/cc266483.aspx),得到完全的dump文件。
explorer hang住了,我看了一下这个进程,和临界区,想看看是不是有死锁等待: 1: kd> .process /r /p 88e10cc0 (explorer.exe) Implicit process is now 88e10cc0 Loading User Symbols ........... 1: kd> !ntsdexts.locks
CritSec ntdll!LdrpLoaderLock+0 at 77247340 WaiterWoken No LockCount 14 RecursionCount 1 OwningThread c40 EntryCount 0 ContentionCount 31 *** Locked
CritSec SHELL32!CMountPoint::_csDL+0 at 7660a0f0 WaiterWoken No LockCount 0 RecursionCount 1 OwningThread e54 EntryCount 0 ContentionCount 6 *** Locked
CritSec WINMM!WavHdrCritSec+0 at 73388f40 WaiterWoken No LockCount 0 RecursionCount 1 OwningThread de0 EntryCount 0 ContentionCount 0 *** Locked
CritSec WINMM!SoundCritSec+0 at 73388f28 WaiterWoken No LockCount 0 RecursionCount 1 OwningThread de0 EntryCount 0 ContentionCount 0 *** Locked
CritSec +3b434e8 at 03b434e8 WaiterWoken No LockCount 0 RecursionCount 1 OwningThread de0 EntryCount 0 ContentionCount 0 *** Locked
CritSec +3b436c8 at 03b436c8 WaiterWoken No LockCount 0 RecursionCount 1 OwningThread de0 EntryCount 0 ContentionCount 0 *** Locked
CritSec +429d838 at 0429d838 WaiterWoken No LockCount 0 RecursionCount 1 OwningThread e18 EntryCount 0 ContentionCount 0 *** Locked
1: kd> !cs 77247340
-----------------------------------------
Critical section = 0x77247340 (ntdll!LdrpLoaderLock+0x0)
DebugInfo = 0x77247540
LOCKED
LockCount = 0xE
WaiterWoken = No
OwningThread = 0x00000c40
RecursionCount = 0x1
LockSemaphore = 0x228
SpinCount = 0x00000000
1: kd> dt _RTL_CRITICAL_SECTION 77247340 ntdll!_RTL_CRITICAL_SECTION +0x000 DebugInfo : 0x77247540 _RTL_CRITICAL_SECTION_DEBUG +0x004 LockCount : -58 +0x008 RecursionCount : 1 +0x00c OwningThread : 0x00000c40 +0x010 LockSemaphore : 0x00000228 +0x014 SpinCount : 0
1: kd> dt _RTL_CRITICAL_SECTION_DEBUG 77247540 ntdll!_RTL_CRITICAL_SECTION_DEBUG +0x000 Type : 0 +0x002 CreatorBackTraceIndex : 0 +0x004 CriticalSection : 0x77247340 _RTL_CRITICAL_SECTION +0x008 ProcessLocksList : _LIST_ENTRY [ 0x7724ab08 - 0x77247220 ] +0x010 EntryCount : 0 +0x014 ContentionCount : 0x31 +0x018 Flags : 0 +0x01c CreatorBackTraceIndexHigh : 0 +0x01e SpareUSHORT : 0
!address看不到: 1: kd> !address 7724ab08 unable to resolve nt!MiSessionViewStart
以上似乎没有看出什么端倪。77247340比较可疑,LockCount有14个,似乎是争用比较多,看了一下它的theard c40:
1: kd> !thread -t c40 Looking for thread Cid = c40 ... THREAD 89218cb8 Cid 0a88.0c40 Teb: 7ffab000 Win32Thread: fd7e7178 WAIT: (Executive) KernelMode Non-Alertable 9a6f6054 NotificationEvent IRP List: 893642e0: (0006,0244) Flags: 00000884 Mdl: 00000000 88c7da28: (0006,0244) Flags: 00000884 Mdl: 00000000 8926b1f0: (0006,0244) Flags: 00000884 Mdl: 00000000 8937d740: (0006,0244) Flags: 00000884 Mdl: 00000000 89396db8: (0006,0244) Flags: 00000884 Mdl: 00000000 Not impersonating DeviceMap 9cc08008 Owning Process 88e10cc0 Image: explorer.exe Wait Start TickCount 16468 Ticks: 32863 (0:00:08:32.666) Context Switch Count 121 UserTime 00:00:00.015 KernelTime 00:00:00.031 *** ERROR: Symbol file could not be found. Defaulted to export symbols for IMJPAPI.DLL - Win32 Start Address IMJPAPI!DllUnregisterServer (0x6cb50928) Stack Init 9a6f7fd0 Current 9a6f5f20 Base 9a6f8000 Limit 9a6f5000 Call 0 Priority 13 BasePriority 8 PriorityDecrement 34 IoPriority 2 PagePriority 5 ChildEBP RetAddr Args to Child 9a6f5f38 836a7a58 89218cb8 00000000 807c2120 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4]) 9a6f5f70 836a6113 89218d78 89218cb8 9a6f6054 nt!KiSwapThread+0x266 9a6f5f98 836a024f 89218cb8 89218d78 00000000 nt!KiCommitThreadWait+0x1df 9a6f6014 8bd6554f 9a6f6054 00000000 00000000 nt!KeWaitForSingleObject+0x393 9a6f607c 8bd6154a 8bd614aa 86850468 00000003 fltmgr!FltpPostSyncOperation+0x8d (FPO: [Non-Fpo]) 9a6f60a0 8bd5e765 86850468 85f21408 86850468 fltmgr!FltpGetNormalizedFileName+0x5e (FPO: [Non-Fpo]) 9a6f60b8 8bd48773 86850468 00000000 86850468 fltmgr!FltpCreateFileNameInformation+0x81 (FPO: [Non-Fpo]) 9a6f60d8 8bd488c7 87e6d3c0 00000000 00000000 fltmgr!HandleStreamListNotSupported+0x125 (FPO: [Non-Fpo]) 9a6f6108 8bd48fa3 c00000bb 00000001 00000000 fltmgr!FltpGetFileNameInformation+0xc7 (FPO: [Non-Fpo]) 9a6f6130 8bd90636 00f21410 00000101 968aec54 fltmgr!FltGetFileNameInformation+0x12b (FPO: [Non-Fpo]) WARNING: Stack unwind information not available. Following frames may be wrong. 9a6f6198 8bd89d7e 85f21410 00000101 968aec54 syscow32v+0xb636 9a6f61f8 8bd42aeb 85f21410 9a6f6218 9a6f6244 syscow32v+0x4d7e 9a6f6264 8bd459f0 9a6f62a8 893642e0 00000000 fltmgr!FltpPerformPreCallbacks+0x34d (FPO: [Non-Fpo]) 9a6f627c 8bd591fe 9a6f62a8 8bd5cf3c 00000000 fltmgr!FltpPassThroughInternal+0x40 (FPO: [Non-Fpo]) 9a6f6290 8bd598b7 9a6f62a8 893642e0 893644f8 fltmgr!FltpCreateInternal+0x24 (FPO: [Non-Fpo]) 9a6f62d4 83675567 881f76b8 87e6d008 8936451c fltmgr!FltpCreate+0x2c9 (FPO: [Non-Fpo]) 9a6f62ec 8bd9bbd1 89364500 893642e0 890bcc08 nt!IofCallDriver+0x63 9a6f6310 8bd9d1ec 881f7800 893642e0 9a6f63c0 flyfs+0x1bd1 9a6f632c 8bd9d4f3 881f7800 0000001b 892211f0 flyfs+0x31ec 9a6f63e0 8bd9df60 881f7800 893642e0 9a6f6400 flyfs+0x34f3 9a6f63f0 8bd9df8a 881f7800 893642e0 9a6f6418 flyfs+0x3f60 9a6f6400 83675567 881f7800 893642e0 8922124c flyfs+0x3f8a 9a6f6418 838795f5 b28c74f3 9a6f65c0 00000000 nt!IofCallDriver+0x63 9a6f64f0 8385a1a7 876f03a0 87f8df78 87e76468 nt!IopParseDevice+0xed7 9a6f656c 83880215 00000000 9a6f65c0 00000240 nt!ObpLookupObjectName+0x4fa 9a6f65cc 83878573 9a6f6750 85f8df78 00000000 nt!ObOpenObjectByName+0x159 9a6f6648 838afd59 9a6f6794 00120089 9a6f6750 nt!IopCreateFile+0x673 9a6f66a4 83801c27 9a6f6794 00120089 9a6f6750 nt!IoCreateFileEx+0x9e 9a6f6700 8bda07c7 9a6f6794 00120089 9a6f6750 nt!IoCreateFileSpecifyDeviceObjectHint+0x59 9a6f67a0 8bd9d593 00000050 9a6f6850 881f7800 flyfs+0x67c7 9a6f6858 8bd9df60 881f7800 88c7da28 9a6f6878 flyfs+0x3593 9a6f6868 8bd9df8a 881f7800 88c7da28 9a6f6890 flyfs+0x3f60 9a6f6878 83675567 881f7800 88c7da28 89282c2c flyfs+0x3f8a 9a6f6890 838795f5 b28c796b 9a6f6a38 00000000 nt!IofCallDriver+0x63 9a6f6968 8385a1a7 876f03a0 85f8df78 88cb7448 nt!IopParseDevice+0xed7 9a6f69e4 83880215 00000000 9a6f6a38 00000240 nt!ObpLookupObjectName+0x4fa 9a6f6a44 83878573 9a6f6b6c 85f8df78 83759400 nt!ObOpenObjectByName+0x159 9a6f6ac0 838afd59 8903c0d0 00100001 9a6f6b6c nt!IopCreateFile+0x673 9a6f6b1c 8bd608e5 8903c0d0 00100001 9a6f6b6c nt!IoCreateFileEx+0x9e 9a6f6bb4 8bd60c59 00000000 00000000 0000fffe fltmgr!FltpExpandFilePathWorker+0x167 (FPO: [Non-Fpo])
看stack trace似乎也没有看到什么,nt!KeWaitForSingleObject等待,但是不清楚下步该怎么分析了,张老师能帮忙看一下么???这个是不是死锁?如果不是,如何分析呢?
|
|
|
IP 地址: 已记录
|
报告
|
|
|
|
2009-08-06, 13:11 下午
|
Da Vinci
注册: 2008-11-03
发 贴: 38
|
|
|
因为机器不在手边,没办法调试,只好用dump文件看系统状态,而且很可能是内核驱动的原因,因为是产品出现的问题。hang住的问题非常难重现,在某些特定机器上会有
|
|
|
IP 地址: 已记录
|
报告
|
|
|
|
2009-08-06, 13:22 下午
|
Da Vinci
注册: 2008-11-03
发 贴: 38
|
|
|
另外,看以下结构: 1: kd> dt _RTL_CRITICAL_SECTION 77247340 ntdll!_RTL_CRITICAL_SECTION +0x000 DebugInfo : 0x77247540 _RTL_CRITICAL_SECTION_DEBUG +0x004 LockCount : -58 +0x008 RecursionCount : 1 +0x00c OwningThread : 0x00000c40 +0x010 LockSemaphore : 0x00000228 +0x014 SpinCount : 0
查看句柄 1: kd> !handle 228 Could not duplicate handle 228, error 6 不清楚为什么会有error 6. 也看不到内核对象。
|
|
|
IP 地址: 已记录
|
报告
|
|
|
|
2009-08-06, 13:48 下午
|
Da Vinci
注册: 2008-11-03
发 贴: 38
|
|
|
看了一下88e10cc0(explorer.exe)进程的各个线程 1: kd> !process 88e10cc0 2 PROCESS 88e10cc0 SessionId: 1 Cid: 0a88 Peb: 7ffdf000 ParentCid: 0a30 DirBase: b9ffe3e0 ObjectTable: 9afd2008 HandleCount: 439. Image: explorer.exe
THREAD 88fbd3c8 Cid 0a88.0a8c Teb: 7ffde000 Win32Thread: fe484238 WAIT: (Executive) KernelMode Non-Alertable 9a635f7c NotificationEvent
THREAD 8908aad8 Cid 0a88.0a9c Teb: 7ffdd000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable 88c839e8 SynchronizationTimer 890846c8 SynchronizationTimer 89081ad0 SynchronizationTimer
THREAD 890b75f0 Cid 0a88.0aac Teb: 7ffdc000 Win32Thread: fd7edb80 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 890fd728 Cid 0a88.0ab4 Teb: 7ffdb000 Win32Thread: fe670660 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 890d6d48 Cid 0a88.0ab8 Teb: 7ffaf000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 890d65c0 Cid 0a88.0abc Teb: 7ffae000 Win32Thread: fe6d2ac0 WAIT: (UserRequest) UserMode Non-Alertable 89011490 SynchronizationEvent 89054da8 SynchronizationEvent
THREAD 89083630 Cid 0a88.0acc Teb: 7ffad000 Win32Thread: ff5654f8 WAIT: (UserRequest) UserMode Non-Alertable 89088ff0 SynchronizationEvent 8904c318 SynchronizationEvent
THREAD 87c46030 Cid 0a88.0bec Teb: 7ffac000 Win32Thread: fe723dd8 WAIT: (UserRequest) UserMode Alertable 8926e3e8 NotificationEvent 87b776d8 NotificationEvent 891224e8 NotificationEvent 88fcfdb8 NotificationEvent 8911a278 NotificationEvent 8916ce60 NotificationEvent 88fd4b40 NotificationEvent 89122548 NotificationEvent 890b89e0 NotificationEvent 890b8638 NotificationEvent 88fad5c0 NotificationEvent 88c656d0 NotificationEvent 890b1400 NotificationEvent 88f9d198 NotificationEvent 890ef480 SynchronizationEvent
THREAD 89218cb8 Cid 0a88.0c40 Teb: 7ffab000 Win32Thread: fd7e7178 WAIT: (Executive) KernelMode Non-Alertable 9a6f6054 NotificationEvent
THREAD 8926d6e0 Cid 0a88.0c44 Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable 890fc670 NotificationEvent 889ad328 NotificationEvent
THREAD 8926fd48 Cid 0a88.0c50 Teb: 7ffa9000 Win32Thread: fd7802a0 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 89258a98 Cid 0a88.0c54 Teb: 7ffa8000 Win32Thread: fd7cf170 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 8927e8b0 Cid 0a88.0c5c Teb: 7ffa6000 Win32Thread: ff58f418 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 8927e030 Cid 0a88.0c60 Teb: 7ffa5000 Win32Thread: fd7d52a0 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 890b4288 Cid 0a88.0c78 Teb: 7ffa4000 Win32Thread: fe486578 WAIT: (WrUserRequest) UserMode Non-Alertable 890ff228 SynchronizationEvent
THREAD 89368998 Cid 0a88.0d04 Teb: 7ffa3000 Win32Thread: fd7cfdd8 WAIT: (WrUserRequest) UserMode Non-Alertable 87b7c298 SynchronizationEvent
THREAD 88e76688 Cid 0a88.0de0 Teb: 7ffa2000 Win32Thread: ff58d008 WAIT: (UserRequest) UserMode Non-Alertable 8924e030 NotificationEvent
THREAD 891c5470 Cid 0a88.0dfc Teb: 7ffa1000 Win32Thread: fd687428 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 891ab628 Cid 0a88.0e00 Teb: 7ffa0000 Win32Thread: ffb566b8 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 88b04998 Cid 0a88.0e18 Teb: 7ff9f000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 891c8d48 Cid 0a88.0e34 Teb: 7ff9e000 Win32Thread: fd6d72a0 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 88aa4030 Cid 0a88.0e54 Teb: 7ff9d000 Win32Thread: ff479830 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 889bcd48 Cid 0a88.0e6c Teb: 7ffa7000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
THREAD 88f7d030 Cid 0a88.05e4 Teb: 7ff9c000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable 890d80f0 SynchronizationEvent
|
|
|
IP 地址: 已记录
|
报告
|
|
|
|
2009-08-06, 16:36 下午
|
Da Vinci
注册: 2008-11-03
发 贴: 38
|
|
|
看了下hang 228: 1: kd> !handle 228 processor number 1, process 85f22920 PROCESS 85f22920 SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000 DirBase: 00185000 ObjectTable: 8cc01c58 HandleCount: 2276. Image: System
Handle table at 8e01a000 with 2444 Entries in use 0228: Object: 87bff830 GrantedAccess: 00000003 (Protected) Entry: 8cc03450 Could not read ObjectType address
1: kd> !object 87bff830 Could not read ObjectType address 1: kd> dt -r _KEVENT 87bff830 nt!_KEVENT +0x000 Header : _DISPATCHER_HEADER +0x000 Type : 0x5 '' +0x001 TimerControlFlags : 0 '' +0x001 Absolute : 0y0 +0x001 Coalescable : 0y0 +0x001 KeepShifting : 0y0 +0x001 EncodedTolerableDelay : 0y00000 (0) +0x001 Abandoned : 0 '' +0x001 Signalling : 0 '' +0x002 ThreadControlFlags : 0x80 '' +0x002 CpuThrottled : 0y0 +0x002 CycleProfiling : 0y0 +0x002 CounterProfiling : 0y0 +0x002 Reserved : 0y10000 (0x10) +0x002 Hand : 0x80 '' +0x002 Size : 0x80 '' +0x003 TimerMiscFlags : 0 '' +0x003 Index : 0y0 +0x003 Processor : 0y00000 (0) +0x003 Inserted : 0y0 +0x003 Expired : 0y0 +0x003 DebugActive : 0 '' +0x003 ActiveDR7 : 0y0 +0x003 Instrumented : 0y0 +0x003 Reserved2 : 0y0000 +0x003 UmsScheduled : 0y0 +0x003 UmsPrimary : 0y0 +0x003 DpcActive : 0 '' +0x000 Lock : 8388613 +0x004 SignalState : -2022748640 +0x008 WaitListHead : _LIST_ENTRY [ 0x876f1870 - 0x94c45988 ] +0x000 Flink : 0x876f1870 _LIST_ENTRY [ 0x58000a - 0x140001 ] +0x004 Blink : 0x94c45988 _LIST_ENTRY [ 0x1580705 - 0x10020140 ]
|
|
|
IP 地址: 已记录
|
报告
|
|
|
|
2009-08-06, 22:47 下午
|
格蠹老雷
注册: 2005-12-19
发 贴: 1,303
|
|
|
explorer并不是什么关键的进程,如果整个系统都失去响应,那么建议还是先从CPU的状态找线索,然后看关键进程,比如CSRSS。死锁基本是肯定的,为什么不执行kdexts.locks,而非要执行用户态的?
|
|
|
IP 地址: 已记录
|
报告
|
|
|
|
2009-08-07, 08:59 上午
|
Da Vinci
注册: 2008-11-03
发 贴: 38
|
|
|
张老师,执行kdexts.locks的结果我没看出什么:
1: kd> !locks **** DUMP OF ALL RESOURCE OBJECTS **** KD: Scanning for held locks.
Resource @ nt!PnpRegistryDeviceResource (0x8379ee80) Exclusively owned Contention Count = 72 NumberOfExclusiveWaiters = 3 Threads: 85fbd370-01<*> Threads Waiting On Exclusive Access: 8911e770 85fb9a70 85f55a70
KD: Scanning for held locks.....
Resource @ 0x87b199f4 Shared 12 owning threads Threads: 88cfbd48-02<*> 88cfb778-02<*> 88a20740-02<*> 88cfba60-02<*> 88d06d48-02<*> 88c807c8-02<*> 89218cb8-03<*> 88d06a60-02<*> 88fbd3c8-03<*> 8938fbe8-01<*> 89127d48-02<*> 8912cd48-02<*> KD: Scanning for held locks.................................................................................................................................................................................................................................................................................................................................................................................................................................... 13577 total locks, 2 locks currently held
资源87b199f4: 1: kd> dt _ERESOURCE 87b199f4 nt!_ERESOURCE +0x000 SystemResourcesList : _LIST_ENTRY [ 0x87b199bc - 0x87b0c02c ] +0x008 OwnerTable : 0x89016898 _OWNER_ENTRY +0x00c ActiveCount : 1 +0x00e Flag : 0 +0x010 SharedWaiters : (null) +0x014 ExclusiveWaiters : (null) +0x018 OwnerEntry : _OWNER_ENTRY +0x020 ActiveEntries : 0xc +0x024 ContentionCount : 0 +0x028 NumberOfSharedWaiters : 0 +0x02c NumberOfExclusiveWaiters : 0 +0x030 Address : (null) +0x030 CreatorBackTraceIndex : 0 +0x034 SpinLock : 0
另外,该如何从CPU的状态入手呢?
|
|
|
IP 地址: 已记录
|
报告
|
|
|
|
2009-08-07, 12:45 下午
|
格蠹老雷
注册: 2005-12-19
发 贴: 1,303
|
|
|
这样的问题靠“遥控”是不大可能解决的,如果想让大家帮忙,那么最好上传到某个FTP,或者压缩一下发到信箱中。
|
|
|
IP 地址: 已记录
|
报告
|
|
|
|
2009-08-07, 13:50 下午
|
Da Vinci
注册: 2008-11-03
发 贴: 38
|
|
|
张老师,这个dump文件很大,压缩之后有400MB,是完全内存转储,不压缩要近3GB大小,信箱放不下。网络硬盘似乎也不行...
|
|
|
IP 地址: 已记录
|
报告
|
|
|
|
2009-08-07, 14:56 下午
|
Da Vinci
注册: 2008-11-03
发 贴: 38
|
|
|
张老师,我把文件压缩成多个文件包,发到您邮箱当中,不知可否?
由于太大,所以只好这样。
另外,dump文件的操作系统版本是win7的7201,不知您是否有调试符号
|
|
|
IP 地址: 已记录
|
报告
|
|
|
|
高端调试 » 软件调试 » Windows内核调试 » Re: 一个系统hang住的问题,请教各位老师
|
|
|
|
|
|