Production debugging ASP.NET applications - PowerPoint by uqg17706

VIEWS: 122 PAGES: 34

									Production debugging
ASP.NET web applications
                   Alex Thissen
                  Trainer/coach
Agenda
•   Why production debugging is hard
•   Debugging process
•   Tools
•   Common scenarios and ASP.NET specifics
    • Crashes
    • Memory leaks
    • Hangs
• Questions and answers
Introducing…
• Class-A
  –   Kennisprovider
  –   Microsoft development
  –   Training & Coaching
  –   http://www.class-a.nl
• Alex Thissen
  – Trainer/ coach
  – http://blog.alexthissen.nl
Challenges of production environment




                             Errors you
   Limited    No Visual         never
                                           High stakes
  access to   Studio to     experienced
                                          and pressure
environment    debug           during
                               testing

                     Challenges
Production debugging process


 Prepare   Reproduce     Observe       Analyze   Fix




              If necessary, reproduce again


• Focus is on ASP.NET applications
   – But, most applies to other apps as well
Getting started                          Prepare


• Create debug build if possible
    – Have symbols ready for debugging
•   Get access to production machine
•   Deploy tools on machine
•   Learn tooling
•   Know how ASP.NET
    and CLR work
Essential tooling                          Prepare


• Debugging Tools for Windows 6.11
  – Debuggers: WinDbg, KD, CDB
  – Scripts: ADPlus.vbs
• Debug Diagnostics 1.1
• Stress and load testing
  – TinyGet (IIS 6.0 Resource Kit)
  – Visual Studio 2008/2010 Test Edition
Reproducing the problem                    Reproduce


• Reduce problem to its essence
  – Eliminate differences and unknowns
  – Find simplest path that leads to bug
• Repeat bug situation
  – Put it under load
  – Make it crash again
• Make it easy to
  reproduce situation
Gathering information                Observe


• Event log
• Application trace and log information
  – Health monitoring
               • Reliability and
                 performance monitor
               • Make a memory dump
Scenario 1: Unexpected restarts
• Symptoms
  • Slow response time
  • Error in browser
  • Loss of session state
• Determine
  • Why caused application restart?
  • Was it our fault or someone else's?
   Unexpected application restarts
External shutdown            Unhandled exception
• IISReset                   • Exception on non-request thread
• Preemptive recycle
• Kill
                        Possible
                         causes

Process exit trigger         Application changes
• Out of memory (OOM)        • Machine/Web.config
• Stack overflow             • Assemblies
• Heap corruption
1st chance, 2nd chance exceptions
Worker process (w3wp.exe or aspnet_wp.exe)
                            Common Language Runtime
  Incoming      Handling                         Background
                requests
                            CLR Worker threads
  requests                                       and GC threads

                                 1st chance
                                 exception


                                2nd chance
                                exception
       Global exception
       handler                                CLR Shutdown
                    Two strikes, YOU'RE OUT!
Finding shutdown reason
• Health monitoring
<healthMonitoring>                                       web.config
 </rules>
  <add name="Lifetime"
     eventName="Application Lifetime Events“
     provider="SqlWebEventProvider” profile="Default«
     minInstances="1" maxLimit="Infinite" minInterval="00:01:00" />
 </rules>
</healthMonitoring>

• Investigate _shutdownMessage and
   _shutdownStack fields of HttpContext
Tracking down exceptions
• Strategies
  – Attach debugger and stop on exceptions
  – Look at recent exceptions in hang dump
  – Log exceptions with debugger running
• Tip: disable pool recycling
  – Recycled apps get in the way of finding real
    reason for shutdown
Demo: crash application pool
• Caused by stack overflow
• Do some live debugging
CDB and WinDbg quick overview
• Debuggers for unmanaged code only
• Still very useful for managed code
• Load extension Son of Strike (sos.dll)
  – Make sure it’s for correct .NET framework
  – .loadby sos mscorwks
Scenario 2: Memory leak
• Symptoms
  • Increasing memory usage
• Determine
  • Managed or native memory leak
• Possible causes
  • Static event handler
  • Finalizer thread blocked
  • Forgotten GC handles
Memory dumps
• .dmp files with complete process state
• Useful when debugging is not possible
  – Unable to attach debugger
  – Post mortem analysis of crashes
• 64-bit specifics
  – Dumps get big
  – Different GC algorithm (less collections)
Taking a dump
              • Tools to create a dump
                 1. ADPlus.vbs script:
                    adplus –crash –pn w3wp.exe
                    adplus –hang –p 1337
                 2. Debug Diagnostics 1.1
                 3. Task Manager in Vista and
                    Server 2008
• Make dump at the right time
  – Hang: catch it in the act
  – Crash: JIT or post mortem
                          About classes, methods and objects
                                                        Some part of managed memory
                                Method tables (MT)

                                                                             Managed heap
Method descriptors (MD)




                                   03ef00cd: Customer
                            03ef015d
                                                          0840a2ac
                                                          Message: "…"
                                                          InnerException:
                            03ef0264

                                                          0842001a           0841fda3
                                                          Name: "Jansen"     Message: "…"
                                  04ab0712: Exception     City: Hedel        InnerException: null
                            04ab0720
                                                          084200fd           084200fd
                                                          Name: "Stelt"      Name: "Stelt"
                            04ab085f
                                                          City: Ridderkerk   City: Ridderkerk



                           Loaded managed code                   Objects on heap
      Memory in Windows
Virtual address space                (Max: 2GB on 32 bit, 4GB on 64 bit)

         Working set      CLR Managed        Native code
                            memory            memory
         (recently used
         memory pages)


                            Private bytes (not shared)
Demo: memory leak
• Caused by rooted references
Scenario 3: "Hanging" application
• Symptoms
  – Slow response times
  – Near 100% CPU usage
• Determine
  – Who's using up
    CPU cycles?
Reasons for hangs




 High CPU hangs        Low CPU hangs
 • Infinite loop       • Deadlocks caused by
 • High % time in GC     Lock, WaitOne or
 • Busy Server           critical Section
                       • Waiting for external
                         resources
Find out what's causing hang
• .time for time busy and !runaway
• Dump native and .NET stacks of threads
  – ~* kb 2000 and ~* e !clrstack
  – What are threads doing?
• Find any deadlocks with !syncblk
  – SosEx might help with other blocking kinds
Heaps of garbage
                                               N-core processor

                     Large Object
    Heap 0                                Heap #N         LOH #N
                     Heap (> 85k)   …




       1     2   3     5 6
                         4         12
                              7 11 8 9 10 13     14 15   16




Gen 2
                                                         Sweep
                                                         Compact
                                                         Mark
        Gen 0
    Gen 1
     • Generational garbage collection
     • Unreachable vs. rooted reference
Demo: High CPU hang
• Caused by spending a lot of time on
  garbage collection
• Load test application
• If I feel lucky: remote debugging
Additional tools
•   SOSex WinDbg extensions:
•   MDbg (C# managed debugger)
•   SOSAssist
•   CLR Profiler
•   Global Flags (part of Debugging Tools)
Top tips
• If you do have Visual Studio 2005/2008
  on production server:
  – You can load SOS during debugging!
  – Managed and unmanaged debugging
• Make sure you have symbols
  – No hassle demystifying call stacks
• Open dump files on same architecture
  machine
Next steps…
• Practice debugging
  • Do labs on Tess Ferrandez' blog
  • Don't hesitate to look at production machines
• Go and see my colleagues as well
  • Frits Ankersmit: DSDM voor kleine projecten
  • Anko Duizer: Windows Azure for Architects 1 & 2
  • Astrid Hackenberg: Working with Web Services in BizTalk
    Server
  • Dennis van der Stelt: If you build it, you'll ship it
  • Alex Thissen: Oslo overview & M language
Review
•   Be prepared: learn your debugging stuff
•   Let debugging process guide you
•   Symbols for effective debugging
•   Use your instinct now and again
•   Don't be afraid to debug production apps
•   Get the debug tooling
Resources
• Blogs:
  Tess Ferrandez, Doug Stewart, Tom Christian,
  Carlo Cardella, Johan Straarup, Jim Robbins,
  Volker von Einem
• WinDbg Help file
• Presentations
  – Common ASP.NET production issues and how to troubleshoot
    them with WinDbg
  – ASP.NET Hang analysis with VSTS Test part 1 and 2
  – Hardcore production debugging of .NET applications
Questions? Answers! Discussion!!!
Thanks for attending
     Happy debugging!

								
To top