Sniffing for crashes (09 Jan 2008)

If only I could remember, that is

Well, I remember the basic approach, and I even took some notes back then which I will regurgitate here and now. However, don't expect step-by-step instructions.

My goal was to take control of the crash reporting process in my application. When an application crashes, Microsoft's official recommendation is that it should not try to catch the fatal exception, but instead simply bail out, let Windows perform its crash reporting rites, and then terminate. For various reasons, we needed more control over the process, and so I set out on a discovery tour through the Windows Error Reporting APIs which Microsoft introduced in Windows Vista. The details of this epic saga can be found at:

• The end is nigh (for my process)
• Crashing with style on Vista
• Crashing with style on Vista, part II
• Think globally, dump locally
• Don't dump Vista just yet; let Vista do the dumping!
• Elementary, my dear Watson!

For a long time, I simply couldn't get crash reporting to work as I expected it to. I had pretty much exhausted most of the official documentation, so I needed to dig deeper.

When an application crashes and the user agrees to send the crash data to Microsoft, Windows contacts Microsoft's Watson servers. For some reason, my crash reports weren't accepted there, while the usual plain vanilla crashme.exe could successfully dump its debris to Microsoft's servers. Hence, the idea was to look at the network traffic and find the differences between my crash reports and those produced by the proverbial crashme.exe.

I could have run the usual network sniffing suspects to decode LAN traffic including all the email exchanged between my boss and his boss, of course. But there is an even easier and less controversial approach: For corporate environments, where admins often need more control over the crash reporting process, Microsoft introduced corporate error reporting (CER), where crashing systems can contact a local server rather than sending all those confidential access violations to Microsoft.

There are registry entries to set the server name and port for corporate error reporting:

• HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Windows Error Reporting\CorporateWERServer: Name of the local crash reporting server
• HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Windows Error Reporting\CorporateWERPortNumber: Port number to be used for communication

On my Vista system, I modified CorporateWERServer so that it referred to my laptop. I did not set the port number explicitly; using procmon, I found that the default port is 1273.

On my laptop, I installed netcat and had it listen to input from port 1273 (nc -l273 or something like that). Once the port was open, Vista started to send HTTP POST requests to it - so the CER server really is a specialized HTTP server listening to port 1273! Here's a typical request (slightly polished and anonymized) following a crash in a sample app I was writing back then:

POST /stage2.htm HTTP/1.1
User-Agent: MSDW
Host: mylaptop:1273
Content-Length: 1110
Connection: Keep-Alive

<?xml version="1.0" encoding="UTF-16"?>
<WERREPORT xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <MACHINEINFO machinename="myvistabox" os="6.0.6000.2.0.0.256.16" lcid="1033"/>
   <USERINFO username="myvistabox\clausb"/>
   <APPLICATIONINFO appname="werapitest.exe" apppath="C:\tmp\werapitest.exe"/>
   <EVENTINFO reporttype="1" eventtime="128267449141252896" eventtype="werapitest (eventType)" 
      friendlyeventname="werapitest (friendly event name)" eventdescription="Critical runtime problem"/>
   <SIGNATURE>
   </SIGNATURE>
</WERREPORT>

By comparing this kind of payload with the traffic generated by a plain vanilla crashme.exe program, I could experiment with the various WER APIs and settings until I had finally figured out how to use them. Without crash report sniffing, I'd probably still be experimenting...