
Miscellaneous Watcom Debugger Issues
-------------------------------------------------------

Debug format
===========================
The default WATCOM style information has a limit of 64K local symbol
information per module. That normally isn't a problem, except for C++
with templates or larger apps. 
We've added support for the DWARF format in 10.5 and above.
This format has no size restrictions. 

The rules for dwarf debug format:

- You have to use the same debugging format for all object files linked into
  your program. That is, if you compile one source file with DWARF information,
  they _all_ have to be compiled with DWARF information.
- Use the -g2d switch for dwarf-1 format debug on the cc line.
  If you compile and link separately, be sure to provide -g2d in both cases.
  Or provide the equivalent option to the compiler and linker.
     e.g. -g2 -hd  for compile, de dwarf for link
  We recommend using 'cc' in all cases.
- The wd debugger will automatically figure out that you're using DWARF
  information.

Using WD over tcp
============================
wd needs a license, so the machine running wd must be able to acquire 
a Watcom license. The machine running tcpserv does not need a license.

In order to use TCP/IP to remotely debug a program, you must start the
TCPSERV server program first. For example,

A>tcpserv
Socket port number: 1024
WATCOM TCP/IP Debug Server
Version 10.6
Copyright by WATCOM International ...
Press 'q' to exit

The server program displays an available socket port number on the 
screen.

You also need to know your IP address. This can be
in alphanumeric or numeric form (for example, 
jdoe.watcom.on.ca or 172.31.0.99).

To use the remote TCP/IP server, you must specify the TCP/IP trap
filename to the debugger along with an argument consisting of the
socket port number given by TCPSERV and your IP address. You must
also include the name of the application you wish to debug. For 
example:

B>wd -tr="tcp;1024.jdoe.watcom.on.ca" app
   or
B>wd -tr="tcp;1024.172.31.0.99" app

The TCP/IP remote debug service permits debugging of applications
anywhere on the Internet. However, response varies with the type of
connection involved. 
The 'app' must be the path to the real binary being debugged. The protocol
will not automatically download the executable to the target.

Source paths
======================
You can use @r and @l in your 'set source pathname' to tell the debugger 
where the source can be found.

@l means try to find the file from the local machines perspective
@r indicates that the files should be searched from the remote machines
   perspective

For example, suppose a host and target setup, h and t. Using tcpserv as the
debug medium.

If my source is on the h machine, then in the 'source path' debugger window 
I could set:

	set source @l/test1     and the h machine would search /test1 for src 
    set source @r/targetsrc and the t machine would search /targetsrc for src

Debug Slib*
======================
>We have an application that occasionally core dumps (SIGSEGV) in Slib32.
>We have discovered that when this occurs, there is no ability to get any
>sort of a stack trace ("Code/Calls") from wd.

Slib32 isn't compiled with the 'generate traceable stackframes' option.
Secondly Slib32 involves a few far calls which may cause wd
problems.  The best approach is to probe the stack.
Your program's code segment is '4+privity', usually 7.  Look up the
stack for a '7', then disassemble at the value immediately following
it -5 to see if it's a call that can end up in the Slib.

errno in compile/debug
=======================

>Is it possible to see what errno is set to from the debugger?

errno is defined to be (*__get_errno_ptr()), so in wd you could:
call __get_errno_ptr()
[responds back with a value]
? *(int *)value 

Or:
int *myerr

void main(void)
{
   myerr = __get_errno_ptr();
   ...
}

The assignment must be in main - do not make it global. This allows
you to set a watch on it in the debugger.

Overriding Symbol and timestamp information in wd
==================================================
wd -trap=pmd\;i :<sym_file> <pmd_file>

The "\;i" tells the debugger to ignore a time stamp mismatch and the
":<sym_file>" says the name of the file where the symbolic information
is to be found is <sym_file>. The file can be an executable with the
symbolic information attached to the end of it, or a file containing
just the symbolic information after it's been separated by wstrip
or the linker "option symfile" directive.

Remember to either quote or escape (\) the ; symbol.
e.g. wd -tr="pmd;i"  is equivalent from the shell's perspective to 
     wd -tr=pmd\;i

Debugging an Interrupt Handler
==============================================================================
There are several approaches to debugging an interrupt handler. Remember that
in QNX your interrupt handler code is far called into by the kernel. This
means that source level debugging (tracing through in wd) is not possible
for an interrupt handler.
How then do I see what is going on in my interrupt handler?

1. From wd you can see the values of global variables that are modified
   by your interrupt handler.
   Let's look at a simple example of an interrupt handler:

   volatile unsigned counter;
   unsigned it_count = 100;

   /* The hardware interrupt handler */
   #pragma off( check_stack );
   pid_t far handler()
   {
	   /* Kick a proxy every 'it_count' timer interrupts */
       ++counter;
       if ( counter == it_count ) {
          counter = 0;
          return(proxy);
          }
       return(0);
   }
   #pragma on( check_stack );

   'counter' is a global unsigned int that will be incremented each time the
   handler is run (and then reset to zero when it hits 100)
   If desired, I can display the value of 'counter' in a window, using:
       DBG> Print/window counter

   This variable window will display the current state of the 'counter'
   variable. However, it will NOT update each time the interrupt handler runs.
   The variable window will ONLY update when you step through the application 
   program. This is because the interrupt handler code is not run from the
   application but far called from the kernel (with your application's data
   segment DS set).
   The same information applies to watch points and break points on global
   variables updated by the interrupt handler.
   
2. An easier approach to debugging an interrupt handler might be to use
   Trace() calls.
   Examine the following interrupt handler:
   
   /* The hardware interrupt handler */
   #pragma off( check_stack );
   pid_t far handler()
   {
	   unsigned char c;

	   c = inp(port);        /* read in a byte from a port */
       Trace1( _TRACE_TEMPORARY, _TRACE_TESTING, (int)c);
       return(0);
   }
   #pragma on( check_stack );

   Every time my interrupt handler runs, a trace event will be logged with
   the data read in from the port.

   Before running the program, check to make sure that the following has been
   done:
     - You may get a lot of trace events generated. To increase the size of
       Proc's trace buffer, make a new Operating System image with a larger
       -T option (maximum is -T 63 which is a buffer of 63Kbytes).
       See the section on Making a Custom OS Image in the User's Guide
       (chapter 11).
   
     - In the above example, we use a severity level of _TRACE_TESTING which
       is 7. Normal trace logging will only save trace events of severity 
       5 and higher (0 is the highest).
       To change this, use the 'tracectrl' program.
         e.g. tracectrl -s 7      You must be root to do this.
       You must change the default each time the computer is booted.

     - The trace events will be logged with a code of _TRACE_TEMPORARY which
       is 0xfffff (see /usr/include/sys/trace.h and /usr/include/sys/tracecod.h)

     Now you can run your interrupt handler program. Let it run for a while
     and then stop it. 
     To see the trace events in raw form, type 'traceinfo' and look for the
     trace events with a code of 0xfffff
     (you could type 'traceinfo -M 0xfffff' to see only the trace events that
      you have generated).
     Millisecond timestamp resolution is available. Use the -m option on
     traceinfo.

     If you want to get even more fancier, you can make a trace format string
     to match the trace events that you are generating.
     For example, suppose I make a file called 'tracetesting' that contains
     the lines:

     #major 0xfffff     /* Trace Temporary */
	     0 Input from port is 0x%x

     Then I can tell traceinfo to use this file for processing the trace
     event.
     e.g. traceinfo -M 0xfffff -m -e tracetesting

     Output will look something like:
      
     Feb 15 11:48:53.189 7 fffff000 Input from port is 0x29

     Using this approach you can make very sophisticated logs of what is
     happening in your interrupt handler. See the documentation on traceinfo,
     tracectrl and tracelogger (as well as the Trace() functions in the 
     Watcom docs) for more information.

3. The third approach is for those who want to do in-depth diagnostic work.
   It is possible to invoke the low-level Operating System Debugger from
   within your interrupt handler.

   Here's an example:

   void    ll_debug(void);
   #pragma aux ll_debug = "int 0f3h"\
       parm [] modify exact nomemory [];

   /* Please note that the ll_debug(void) pragma is defined in 
      /usr/include/sys/inline.h   ... you should include this header rather
      than define the pragma yourself. It is reprinted above for completeness.
    */

   /* The hardware interrupt handler */
   #pragma off( check_stack );
   pid_t far handler()
   {
	   unsigned char c;

       ll_debug();
       return(0);
   }
   #pragma on( check_stack );

   The pragma ll_debug() is defined in /usr/include/sys/inline.h  There are
   other useful pragmas in this file ... it is worthwhile to scan through
   inline.h.

   Before running the handler program, you must make sure that the Debugger
   is built into your Operating System image.
   Add the following two lines to the bottom of your Operating system boot
   image (found in /boot/build):

    sys/Debugger32
    $ Debugger32

   By default when you boot an OS image that has the debugger built into it,
   the operating system will fall into the debugger 3 times when it boots.
   Press 'g' (for GO) at each of the debugger prompts to continue on.
   If you want to boot your image without dropping into the debugger, put
   a -D option on Proc in your boot/build file.
   e.g.
   sys/Proc32
   $ Proc32 -l 1 -D

   Once you boot up with this new image, you can enter the low-level debugger
   at any time by hitting CTRL-ALT-ESC at the same time.
   From within a program you can enter the debugger by issuing an int f3h call
   (which is what the ll_debug pragma does).

   Run your interrupt handler program. Each time your handler runs, you will
   drop into the low-level debugger.
   *** IMPORTANT ***
   When you are in the low-level debugger, the entire QNX system is halted!
   No other processes will be running at that point.
   See the section on 'Debugger' in the Utilities Reference for a list of
   available commands.

   At this point you can step through the assembly code of your interrupt
   handler, look at memory (using d), read in values from I/O ports using
   i or I etc...
   To continue on until the next interrupt to occur, type 'g'.
   This can be very handy for verifying the behaviour of hardware.

Post Mortem Debugging
============================================================================
   Sometimes a program will run fine for a period of time (days/weeks) and 
   then die abnormally. 
   The problem could be a bad pointer somewhere or a floating point error
   (divide by zero) etc. and can be very difficult to diagnose using normal
   debugging tools.
   In QNX these "very difficult to find" bugs can be diagnosed quite easily
   using post mortem debugging.

   By running a background process called 'dumper' , an image of the program
   that died will be saved in the filesystem.
   Then at a later time I can start wvideo and tell it to process the 'dumped'
   file image. I can see the state of the program at death: variables, where 
   exactly in the C/C++ code the crash occurred etc.

   Let's look at an example:

   Here's a sample piece of code called 'crash.c':

   #include <stdio.h>

   main()
   {
      char buff[200];
      char *p;

      p = 80000;
      *p = 'a';
      buff[0] = 'b';
   }

   If you compile ( cc crash.c -o crash -g ) and run this program, the program
   will terminate with a SIGSEGV violation when it tries to write an 'a' into
   the memory location pointed to by p (which was accidentally set to 80000).
   If I start the 'dumper' program before I run the program again, then an
   image of the program at time of death will be saved in a file called 
   'crash.dmp'.
   See the section on 'dumper' in the Utilities Reference for more information.

   Now, when I have this 'crash.dmp' file, I can see exactly in the C code
   where the segment violation occurred.
   Copy the 'crash.dmp' file into the directory where the crash.c source is.
   (this is easiest; you can also use the 'Set source' command in wd).

   Inside this directory, start up wd with the -tr=pmd option and tell
   it to diagnose the dump file.

   e.g.   wd -tr=pmd crash.dmp

   The debugger will place a highlight directly over the offending source line.
   At this point you can do some post mortem debugging:
           - print out the contents of variables (e.g. 'Print p' to see the
             value of the bad pointer).
           - type 'show calls' to see how you got to where you are...
           - type 'mix' to see the associated assembly code/stack if necessary.
             etc.
               
Debugging a Running Process
============================================================================
Sometimes a program will be running and not behaving the way I want it to.
It can be handy to 'attach' to the running process and see exactly in the C
code where it is executing.
(maybe the program is looping for example)

In QNX, to debug a running process, follow the steps below:

     - type 'sin' and write down the pid of the process you want to
       debug.  In this example I'll assume the pid is 2542.

     - cd to the directory where the source for the program is kept.
       e.g. cd /work/src

       (alternatively if you do not want to cd to the directory, you can
        use the 'Set Source' command in wd to tell it where to look for
        your source file. I find it's easier to cd to the source directory).

     - type 'wd pid'  where pid is the process id.
          e.g. wd 2542

     - wd will run ... now I can see exactly where in the source I am.
       If you are in the shared library then you won't see source... If you
       don't see source, you can use Trace to step through the shared 
       library call until you get back to your application code.
       As well, you can type 'show calls' to see how you got to where you
       are.

Debugging a remote process also works across the network. This can be handy
for debugging target systems.
For example, suppose my target system is node 2 and my development system
is node 1.
If process 3544 is running on node 2, and I am sitting at a console on
node 1 I can type:

         wd //2 3544

and debug the process from across the network.

Using Sparse Dump files
===================================================
Sparse dumps don't dump out pages that aren't mapped in, which can
greatly reduce the size of a dumpfile for some programs.

You must use the -s option to dumper to generate sparse dump files.

If you use -s to dumper, you must also do "wd -tr=sparse" to handle the
dump file rather than "wd -tr=pmd" as wd needs a different trap file to
know how to handle sparse dumps instead of regular dumps.

