Troubleshooting Guides

Note : This page and the documentation linked to it is under construction.

Table of Contents

• Linux Probing Package Troubleshooting
• TSC-Based Clock Troubleshooting
• TSC-Linux Troubleshooting
• TSC-RT-Linux Troubleshooting

Linux Probing Package Troubleshooting

If you are relying on ICMP replies to be generated due to UDP packages and you notice that packets are being regularly dropped on a host that you have administrator priviledges over and is not running a firewall, you may want to investigate the kernel ICMP rate limiting by typing: cat /proc/sys/net/ipv4/icmp_ratelimit. Setting this value to 0 will prevent any ICMP rate limiting, this can be achieved by typing the following: echo 0 > /proc/sys/net/ipv4/icmp_ratelimit

TSC-Based Clock Troubleshooting

The TSC based packet sender requires some way to convert the TSC counter values into timestamps. This is achieved by using the programs tscskewd(8), which is used to continually estimate the current conversion constant. It should be used in conjunction with a primary NTP server in order to assure maximum accuracy of the conversion constant. The program readtscskew(1), is also provided to read the current count, time and offset values being reported by tscskewd. However, the offset value is not currently set to anything meaningful. If measurements using the same clock are used, or if time differences (each difference being taken using a single clock) are manipulated, then this will not matter as the offset values would cancel in any case.

TSC-Linux Troubleshooting

The Linux 2.4.14-tsc modified kernel includes the modified 3c59x_tsc driver which timestamps in the driver. These timestamps are then more accurate than the conventional linux timestamping, which is carried out in the device independent code called after the driver. In the case you don't have a network card compatable with the 3c59x series, the kernel will still produce TSC-based timestamps, they are just not as accurate.

Note: A modified driver for accurate time stamping is only provided for 3c59x type cards at this time. If you do not have a 3c59x type card then you can view the changes to the 3c59x_tsc.c source code and modify your NIC driver accordingly. We do however have a rtl8139_tsc pre-release driver available on request.

An example kernel config file for 2.4.14-tsc is given here: config-tsc-example. To use this file copy it to .config in the kernel source directory and then run make menuconfig to modify the options according to your system hardware.

Currently the TSC modifications are only available for the 2.2.14 and 2.4.14 kernels, which do not have ext3 support inbuilt. For those of you who have ext3 file systems (e.g. RedHat => 8.0 users), there is a patch which adds ext3 support to the 2.4.14-tsc kernel. Alternatively you can just mount it as ext2.

The patch can be applied by issuing the following command:

gunzip < ./ext3-2.4.14.patch.gz | patch -p1

Once you have patched the 2.4.14-tsc kernel, you will need to run make menuconfig and select EXT3 and JBD debugging support in the Filesystems menu.

Another point to take care of when compiling the 2.4.14-tsc kernel is to check the version of the compiler you are using. The version of gcc that comes with RedHat 8.0 is gcc-3.2-7. We found gcc-2.95.3 was able to compile the 2.4.14-tsc kernel, but had trouble when using gcc-3.2-7. If you are using RedHat => 8.0 you can download the compat-gcc package, which will make gcc-2.96 available as "gcc296". In order to use that version of gcc instead of gcc-3.2-7 you will need to change the variable HOSTCC in the Makefile from gcc to gcc296.

TSC-RT-Linux Troubleshooting

In order to install the 2.2.14-rtl2.2 kernel you will need to follow the following steps. Note that when you are compiling the 2.2.14-rtl2.2 kernel it requires that all of the symbolic links that are mentioned exist, without them the compilation will fail.

The installation requires the compilation of a number of programs, first you need to compile the 2.2.14 kernel to be used with the real-time modules, then you need to compile the real-time modules and finally you need to compile the RTNet code.

All commands you need to type are in this color.

1. Download the 2.2.14-rtl2.2 kernel package and save it to the directory /usr/src.
2. cd /usr/src
3. tar -xzvf rtlinux-2.2.tar.gz (this will untar the package into the directory /usr/src/rtlinux-2.2)
4. Now you need to make a symbolic link from the rtlinux-2.2 directory to one called rtlinux.
   This can be done by running ln --symbolic rtlinux-2.2 rtlinux in the /usr/src directory.
5. Now you need to compile the 2.2.14 kernel, cd /usr/src/rtlinux/linux.
6. If you have a .config file already then copy it into this directory and then run make menuconfig.
   When configuring, you must enable loadable module support and the kernel module loader.
   You must also enable Ethernet 10/100Mbit and any NIC drivers that are you are going to use must be compiled as modules.
7. make dep;make;make modules;make bzImage;make modules_install
8. Now you need to copy the bzImage file to your /boot/ partition:
   cp arch/i386/boot/bzImage /boot/vmlinuz-2.2.14-rtl2.2.
9. Check /etc/lilo.conf and modify it to add the new image that you just copied to the /boot/ partition, rtlinux2.2. For example:

        An example lilo.conf:
        ------------------------------------------
        boot=/dev/hda
        map=/boot/map
        install=/boot/boot.b
        prompt
        timeout=50
        image=/boot/vmlinuz-2.2.5-15
               label=linux
               root=/dev/hda5
               read-only
        image=/boot/vmlinuz-2.2.14-rtl2.2
               label=rtlinux22
               root=/dev/hda5
               read-only              
        -------------------------------------------
   

10. modify /etc/modules.conf and check you have the NIC that you are using in that file. For example we use the 3c59x driver so our /etc/modules.conf has "alias eth0 3c59x".
11. Now run lilo to check if everything is O.K. lilo -v.
    If there are no errors reported then reboot and load the new kernel. lilo -R rtlinux22; reboot
12. Now you need to compile the rtlinux modules:
    cd /usr/src/rtlinux-2.2
13. make;make install;chmod a+x insrtl;chmod a+x rmrtl
14. To test it run ./insrtl; /sbin/lsmod
    You should see the follow modules in the list: rtl_fifo, rtl_posixio, rtl_sched, rtl_time.

Now you need to install the RTNet package.

1. Download the RTNet-0.9 RT networking package and save it to the /usr/src directory.
2. cd /usr/src
3. ln -fs rtlinux-2.2/linux linux
4. ln -fs rtlinux-2.2 rtlinux
5. tar -xzvf rtnet-0.9.0fix.tar.gz
6. ln -fs rtnet-0.9.0 rtnet
7. cd rtnet; make; make install; make modules_install; make dev
8. We assume that the machine contains two NICs, at least one of which is compatible with the 3c59x driver [making other network cards compatible with the RTNet package is left as an exercise for the reader :-) ]. The second interface (eth1) is going to be used as a real time interface. (for more details see the README file in the rtps package, but before installing that package finish with the installation of rtnet)
   If your current configuration file contains the 3c59x driver, please change it to 3c59x_rt_old. This is the RTNet version of the driver, which if configured under Linux works as a Linux driver while if it is activated with the real time interface configuration command works as a RT NIC.
   Now change /etc/modules.conf to include 3c59x_rt_old
9. Now we can test the setup. Reboot with rtlinux22 kernel
   lilo -R rtlinux22;reboot
10. Load the rtlinux modules: cd /usr/src/rtlinux; ./insrtl.
11. Bring the (non real-time) linux interface eth1 (if it exists at all): ifconfig eth1 down
12. Configure eth1 as a RTNet interface:
    rtifconfig eth1 up 10.0.0.114 0.0.0.0
    where 10.0.0.114 is the IP address of the eth1 interface and 0.0.0.0 is the network mask.
13. Check if it all works. Simply ping the address you gave eth1 from a different host, and then check the console output of the real-time host. e.g:
    tail /var/log/messages
    If you see rt_ip_route_input() adding route it means that RTNet is working.

If you made it this far then your configuration should be working! Now you can start to use the real-time packet sending package to send highly accurate probe steams.