If you have high CPU usage of the System process, this article will help you to track down the offending driver or kernel module. In my case, it was intelppm causing an abnormally large percentage of CPU time in the kernel. Intelppm is a Microsoft Intel power management driver for Intel processors that executes in kernel mode (Hence the high CPU of the System process). I tried to use the Process Explorer thread tab on the System process, but since I didn't have Windbg installed and configured with the Microsoft online symbol server, all I could see was hex.
After searching for ways to look at kernel threads, I found Mark Russinovich's blog post and he suggested Kernrate, a command line kernel profiling tool. Symbol files are not required if Kernrate finds a device driver. After installing Kernrate I ran Kernrate_i386_XP.exe on a Windows 2003 x86 server during the high cpu, then hit Crtl+C to stop profiling when it returned to normal. You can use a tool called xperf for newer x64 systems (Win7/8/2008/2012). The following output is what I found:
Results for Kernel Mode:
OutputResults: KernelModuleCount = 137
Percentage in the following table is based on the Total Hits for the Kernel
Time 49788 hits, 25000 events per hit --------
Module Hits msec %Total Events/Sec
intelppm 33834 33953 67 % 24912378
ntoskrnl 13018 33953 26 % 9585309
win32k 864 33953 1 % 636173
hal 690 33953 1 % 508055
klif 450 33953 0 % 331340
fltmgr 294 33953 0 % 216475
vdtw30 131 33953 0 % 96456
Ntfs 120 33953 0 % 88357
tcpip 93 33953 0 % 68477
Npfs 51 33953 0 % 37551
RDPDD 36 33953 0 % 26507
SCSIPORT 29 33953 0 % 21353
snapman 27 33953 0 % 19880
WDICA 18 33953 0 % 13253
mraid35x 18 33953 0 % 13253
e1000325 15 33953 0 % 11044
CLASSPNP 14 33953 0 % 10308
PartMgr 14 33953 0 % 10308
afd 12 33953 0 % 8835
CtxSbx 9 33953 0 % 6626
termdd 9 33953 0 % 6626
CtxAltStr 8 33953 0 % 5890
NDIS 6 33953 0 % 4417
pdcrypt2 4 33953 0 % 2945
RDPWD 4 33953 0 % 2945
ipsec 4 33953 0 % 2945
TDTCP 3 33953 0 % 2208
TDI 3 33953 0 % 2208
pdrframe 2 33953 0 % 1472
ipnat 2 33953 0 % 1472
disk 2 33953 0 % 1472
rasacd 1 33953 0 % 736
dcdipm32 1 33953 0 % 736
volsnap 1 33953 0 % 736
ftdisk 1 33953 0 % 736
It was obvious that intelppm is the offending driver because it accounted for 67% of the hits in the kernel. I searched for an updated intelppm driver but couldn't find any newer versions and since there hasn't been any changes to the server, I decided to cross my fingers and uninstall the driver. It turns out that intelppm is safe to uninstall even on an Intel system. So far it has been stable. There are two methods for Windows Server 2003/XP:
Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
To work around this issue, follow these steps:
- Click Start, click Run, type regedit, and then click OK.
- Locate and then click the following registry key:
- In the right pane, right-click the Start entry, and then click Modify.
- In the Value data box, type 4, and then click OK.
- Exit Registry Editor.
- Restart your computer.
To work around this issue, run the recovery console by using the CD. Then, select the recovery option.
- Insert the Windows CD in the CD drive. Then, restart the computer.
Note If you are prompted, click to select any options that are required to start the computer from the CD drive.
- When the "Welcome to Setup" screen appears, press R to start the Recovery Console.
- If you have a dual-boot computer or a multiple-boot computer, select the installation that you want to access from the Recovery Console.
- When you are prompted, type the administrator password.Note Press ENTER if the administrator password is blank.
- At the Recovery Console command prompt, type the following command, and then press ENTER:
- To exit the Recovery Console and to restart the computer, type exit at the Recovery Console command prompt, and then press ENTER.
If you're as lazy as I am, here are registry files you can download and run instead of performing these operations yourself:
Feel like beefing up your understanding of threads, processes and memory management in Windows? David Solomon, Author of Windows Internals presents at Technet Talks in 2005, focusing on understanding processes, threads, memory management in Windows. It was so good that I created it’s own post
Processes and Thread Troubleshooting - Part 1
Processes and Thread Troubleshooting - Part 2
Understanding and Troubleshooting Memory Problems - Part 2