ITrace: Instruction Trace

• General Description
• Running ITrace
  • Initializing swtrace
  • Collecting ITrace Data
  • Tracing a Java application
  • Capturing ITrace Data
  • Data Reduction with Post
  • Sample swtrace Scenario
  • Perfutil interfaces

General Description

ITrace (Instruction Trace) is a software tracing mechanism that runs on Linux. It uses the software trace facility (swtrace) to allocate a buffer, capture MTE information and write branch trace records. It then uses post to format the swtrace output into callflow reports. ITrace traces through application and kernel code, and it can also trace IBM's Java jitted code.

How ITrace works

1. All pids
2. Only the current pid
3. Only the given pid
4. Only pids greater than the given pid
5. Only pids from the given list of pids

ITrace in association with swtrace, a2n, and post can give you an "arc" report for your application. post can also dump the single step flow if needed (post -ss). ITrace branch tracing works in concert with the software trace facility to trace ring 3 and ring 0 instructions. ITrace uses the normal swtrace buffer and trace formats to record the last branch that executed. It uses the single step trap and branch flags to stop the application flow and write out a callflow hook. This hook includes time stamp, To and From addresses.

ITrace tracing is turned on using two steps:

1. swtrace to setup the buffer and turn on trace options.
2. perfutil is used to invoke ITrace enable/disable and on/off APIs.
   The perfutil APIs use IOCTL calls to our device driver.

It is advisable to "warm-up" the code to be traced. This will reduce the amount of page fault processing that is traced. This is true in both C and Java code where the code path to be examined can be contained in a loop. Also note that with Java, the JIT code path will be traced the first "X" times that a method is jitted. It is advisable to try and keep the trace scenario as simple as possible and limit the amount of pids to trace.

ITrace Hooks

ITrace writes two types of hooks to the swtrace buffer, one that contains branch information for every traced branch and the other when a traced pid changes.
It also uses the swtrace MTE hooks for post-processing name resolution.

Running ITrace

The following section assumes swtrace resides either in a directory listed in the PATH environment variable, or it resides in the current directory.
 

Initializing ITrace

swtrace must be initialized using the swtrace init command. When swtrace is initialized, the required trace buffers (one per processor) are allocated in the kernel address space. The default buffer size is 3MB per processor, a larger buffer will be required to use ITrace.

To initialize swtrace and initialize ITrace:

        swtrace init -s 30
        swtrace it_install
        swtrace disable

Collecting ITrace Data

At this point swtrace has created the trace buffer(s) and initialized everything needed to start instruction tracing. The next step is to start the trace. ITrace is started by making a perfutil call.

    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <stdint.h>
    #include "perfutil.h"

    int main(int argc, char* argv[])>
    {
       ITraceDisable();
       ITraceEnableCurrent();
       ITraceOn();

       // code to be traced
       ...

       ITraceOff();
    }

ITrace and Java

       java -Xrunjprof:itrace <class_name> You can enable/disable pids and turn on/off ITrace directly from your Java application, using Java ITrace APIs.
 

Capturing and Post-processing Trace Data

post is used to format the swtrace.nrm2 into callflow reports. It can produce post.show, arc and ptree files.
The option -arc creates an "arc" file. This is a generic2 file, which is a function call report.
The -arc option along with the -ss option will produce the
function call report along with disassembled instructions.
See post documentation for more detail (also post -?). 

To capture the contents of the trace buffer to file swtrace.nrm2 and post-process it into an arc file, enter:

          swtrace get
          post -arc

ITrace swtrace Command Examples

swtrace init -s 10

This command will allocate a trace buffer of 10 MB per processor and initialize swtrace.

swtrace it_install

This command installs and initializes the ITrace part of swtrace.

swtrace on

This command turns on tracing.

itracegton 1 100

This example uses perfutil to start ITrace to trace all pids.
The second argument is the number of skipped timer ticks and it is currently ignored.

itracegtoff

This command will turn tracing off. Once tracing is off, the user may then process the trace data contained in the trace buffer without having additional data written to the buffer.
 

swtrace get  mytrace.nrm2

This command will extract the trace buffer into the file mytrace.nrm2.
 

post -r mytrace.nrm2 -arc

This command will process the RAW file mytrace.nrm2 and generate an ASCII version in arc.
 

swtrace disable

This will deactivate all trace major codes.
 

swtrace free

This will turn tracing off and free any swtrace buffers.
 

swtrace it_remove

This will remove the ITrace support.
Note that on x86 and x86_64 systems you machine must be rebooted to completely remove ITrace kernel hooks.
 

Sample ITrace Scenario using swtrace

 1.  Initialize swtrace                      swtrace init -s 10
     setting the buffer size to 10 MB per CPU.

 2.  Install ITrace support.                        swtrace it_install
 
 3.  Disable all of the performance trace hooks.    swtrace disable  

 4.  Turn swtrace on.                               swtrace on
 
 5.  Run the desired program 
     with itrace enable/disable, on/off.            execute the program to trace
     (see itracec.c in tools src/swtrace directory).

 6.  When ITrace completes, or at the               swtrace off
     appropriate time, turn off swtrace.
 
 7.  Get a copy of the trace information            swtrace get
     from all processors.

 8.  If all tracing is complete, you could          swtrace free
     now free the trace buffers.

 9.  Generate readable trace data.                  post -arc -ss

 10. Remove ITrace support.                         swtrace it_remove

You may also use the run.itrace script in Dpiperf/bin directory.
 

ITrace perfutil interfaces

Perfutil has the following interfaces for ITrace:

• int ITraceInit(UINT32 type);
  • Initializes the ITrace facility (installs required support for instruction tracing).
    Trace buffers must be allocated prior to invoking ITraceInit() (e.g., swtrace init).
• int ITraceTerminate(void);
  • Terminate the ITrace facility (removes support for instruction tracing.)
    You must re-initialize the facility in order to use it again.
    Trace buffers(s) must be deallocated (e.g., swtrace free).
• int ITraceEnableAll(void);
  • Enables tracing for all processes.
• int ITraceEnableCurrent(void);
  • Enables tracing for the current process.
• int ITraceEnablePid(UINT32 pid);
  • Enables tracing for the given process (pid argument).
• int ITraceEnableGreater(UINT32 start_pid);
  • Enables tracing for processes whose PID is numerically equal to or greater
    than the given start_pid.
• int ITraceEnableRange(UINT32 start_pid, UINT32 end_pid);
  • Enable tracing of all pids in the range [start_pid, end_pid] (including start_pid, end_pid).
• int ITraceEnableList(UINT32 pid_list[], int pid_cnt);
  • Enable tracing of all pids in the list pid_list, which has pid_cnt elements.
• int ITraceSetSkipCount(UINT32 timer_skip_count);
  • Set skip count for ITrace (currently no effect).
• int ITraceOn(void);
  • Start tracing ITrace records.
    Tracing does not start until you issue ItraceOn().
    At that time tracing will begin for all PIDs that have been enabled.
    You should enable all the required PIDs before turning on ITrace.
    You should not invoke any ItraceEnable*() method once you are tracing.
• int ITraceOff(void);
  • Stop tracing ITrace records.
  • You can invoke ItraceOn() to continue tracing.
• int ITraceDisable(void);
  • Disable tracing of all pids.

Java ITrace perfutil interfaces

Trace.class in the JPerf.jar is used to interface with perfutil and has the following ITrace APIs:

• public static native int ItraceInit();
• public static native int ItraceTerminate();
• public static native int ItraceEnableAll();
• public static native int ItraceEnablePid(int pid);
• public static native int ItraceEnableGreater(int startpid);
• public static native int ItraceEnableRange(int startpid, int endpid);
• public static native int ItraceEnableList(int[] pidlist, int pidcnt);
• public static native int ItraceDisable();
• public static native int ItraceOn();
• public static native int ItraceOff();
• public static native int ItraceSetSkipCount(int skipcount);