Active Probing Package - RTLinux apps. ====================================== Version: 0.1 Last change: 08.11.2001 Requires: RTLinux-2.2 and RTNet-0.9.0fix to be installed network interface card ( 3COM: 3c905-TX-M ) ( it is strongly recommended to have two cards in the host, one for the linux level for standard networking, and the 3COM card for real-time networking ) For detailed instructions see: INSTALL-RTLinux and INSTALL-RTNet Please read the whole document before installation. Contents: --------- I. Introduction II. TSC clock III. Probe stream definition IV. Output file formats V. Files in the directory VI. Some comments on Real-Time Networking VII. Installation and test I. Introduction --------------- The aim of this software package is to support active probing. By active probing we mean the injection of artificial probe traffic into the network and the measurement of its characteristics at different points. The basic components of the active probing infrastructure are: Sender --------> NETWORK ----------> Receiver | | | | | | V V Sender Monitor Receiver Monitor The Sender creates and transmits the probe stream. The Receiver is the sink, terminates the probe stream. The Sender Monitor records the packet departure times from the Sender. The Receiver Monitor records the packet arrival times to the Reciever. The arrival and departure timestamps are generated after receiving the whole packet. These timestamps can be generated based on a number of different time sources - clocks. Some of the tools in the package rely on the standard software clock ( gettimofday() ) or in the case of RT-Linux applications on the standard RT-Linux software clock ( gethrtime() ). However due to the many different factors timestamps generated using these clocks are prone to a number of effects which can significantly affect their accuracy. We found that the most reliable and quick way of obtaining a timestamp is simply using the CPU clock cycle register (TSC). II. TSC clock ------------- The accuracy of the timestamps generated using TSC depends on the stability of the CPU clock source and on the accuracy to which the actual clock rate is known, which is used to convert the timestamps to standard time units. Measurements has shown that the stability of the desktop PC CPU clock signal sources is typically better than 0.1PPM. The clock rate can be estimated (clock calibration) using a reference clock with a known accuracy. Most probably the easiest way of the calibration is to use the standard software clock of the system, which is synchronised to an NTP server (preferably stratum 0). The CPU clock rate estimation is then based on counting the clock cycles in an interval of T seconds, the selection of which depends on the accuracy of the reference time source. If e is the accuracy of the reference clock, for example 1ms for an NTP synchorinsed SW clock (assuming synch.to a primary NTP server) the T is app. 6 hours to get an estimate with an accuracy of 0.1 PPM ( 2e/T < 0.1 PPM (1e-7) ). To calibrate your clock according to the above described method use the tscskewd application included in this package. This application continously measures and updates two registers ( count[CPU clock cycle] and time[usec] ) which are used by other applications converting raw TSC timestamps to standard time units. The conversion is done off-line (tscdttodt and stscdttodt), to achieve higher performance and allow higher reliability and accuracy with keeping the raw data. If the accuracy of your synchronised NTP clock is worse than 1ms, average the values reported by the tscskewd utility ( after the 7 hour initialization period one estimate is reported each 96 seconds which is based on a 7 hour period T , 11 estimates are also reported during the initialization period to get a quick but less accurate conversion constant estimates). Note: variations in the estimate can be caused by the changing rate of the TSC clock (which on desktop PCs, according to our experiences, is usually below 0.1 PPM and caused for example by variations in the room temperature ). However we have observed in a number of cases that larger variatins were caused by NTPs attempts to correct for earlier wrong offset corrections. Errors in the estimate due to the second reason can be filtered out simply by taking long term averages. A good TSC conversion estimate remains accurate to 0.1 PPM for many days or even weeks if there are no large changes in the external conditions. For more details on clock accuracy, calibration and synchronistion see our publications. III. Probe stream definition ---------------------------- In general the packets of a probe stream can be defined by their: - packet type - packet size - departure time - time to live ( max. number of hops ) The sender tools in this package are using UDP packets with a packet serial number placed into the first four bytes of the UDP payload. As a result of this the packet size typically can be varied between 32 and 1500 bytes. The time to live field of probes which are meant to reach the receiver should be set to -1. This package is using the '.bin' binary file format to store probe stream definitions. These files are 64 bit floating point (C: double) arrays with num_of_probes x 3 elements (time, size, ttl). New probe stream files can be easily created using Matlab or simple C programs. To create a probe stream using Matlab just follow these steps: 1. p = ones(, 3); 2. p(:,1) = ; 3. p(:,2) = ; 4. p(:,3) = ; // usually just -1 and save p as a double precision binary file using for example 5. pswrite('teststream.bin',p); where pswrite is a simple Matlab function (pswrite.m): function y = pswrite(x,a) % pswrite - creates a binary file of timing information % for rtSend and linuxSend applications % % pswrite('filename',idt_array) % % idt_array - array of inter departure times % fid = fopen(x,'wb'); fwrite(fid,a,'double'); y = fclose(fid); Note: if you are running Matlab on a diffent platform you may encounter problems due to the different 'double' representations (eg. bit order) IV. Output file formats ----------------------- '.stscdt' - tsc based time stamps created by the tscrtps.o and tscrtSend combined Sender & Sender Monitor application '.tscdt' - tsc based time stamps created by the rtps.o and rttscreceiver combined Receiver & Receiver Monitor application '.dt' - the 'standard' timestamp file each timestamp stored in 64 bits in 32bit.32bit format (sec:remainder) the first value of the file is 1 (for compatibility with PSIM) which is followed by num_of_probes timestamps in the format spec. above lost packets are represented as -1 ( 0xffffffffffffffff ) Some Matlab files supporting the analysis of the '.dt' files are included in the package. V. Files in the directory: _________________________ 1secx10.bin - an example probe stream file, sending an initial probe + 10 probe packets with 1sec inter departure times INSTALL-RTLinux - RTLinux-2.2 installation instructions INSTALL-RTNet - RTNet installation instructions Makefile - a makefile README - this file addr2i.c - a simple program to convert hostname or an ip address to the hexadecimal format required by the rtps modules ( program used by the startrtps script) dt_ts.m - Matlab file for post processing '.dt' files pscomm.c, pscomm.h - shared memory communication support for the sender/receiver modules and the sending applications pswrite.m - Matlab file for saving '.bin' files readtscskew.c - simple application reporting the the current TSC conversion constant estimates from tscskewd rtpr.c - RT-Linux packet receiver module based on the RTNet package, packet arrival timestamps are generated in the network interface card's driver (3c59x_rt_old.c in the rtnet package) using the CPU-s clock counter as clock (use rttscreceiver as receiver appl.) rtps.c - RT-Linux packet sender module based on the RTNet package using standard RT-Linux clock based timing (gethrtime()) rtpr.c - RT-Linux packet receiver module based on the RTNet package, packet arrival timestamps are generated in the network interface card's driver (3c59x_rt_old.c in the rtnet package) using the CPU-s clock counter as clock (use rttscreceiver as receiver appl.) rttscreceiver.c - receiver application (Monitor) for the rtpr.o module sendAppl.c - a user level application generating timing information for the packet sender programs (rtps and linuxps) startrt - a shell script to load rtlinux and rtnet modules, it also configures rtnet if file ~/rtnetsetup.txt exists (text file containing the list of rtifconfig commands ) startrtpr - loads the reciever module Usage: startrtpr 7775 startrtps - a simle shell script to load the sender module. Usage: startrtps [tsc] Eg. to load rtps.o: startrtps 10.0.0.4 7775 Eg. to load tscrtps.o: startrtps 10.0.0.4 7775 tsc stoprtpr - removes the receiver module stoprtps - removes the packet sender module stscdttodt.c - when rtps and tscrtps modules are used to send probes 'sent.stscdt' file is created with timestamps of the sent packets. This application converts this file to the general timestamp file format '*.dt' tscdttodt.c - conversion application for the TSC based timestamp file format '*.tscdt' used by the TSC based receiver applications, converts files to the general timestamp file format '*.dt' tscrtps.c - RT-Linux packet sender module based on the RTNet package using the CPU-s clock counter as clock (use tscrtSend as sending appl.) tscsendAppl.c - a user level application generating timing information for the packet sender programs converting them to to CPU clock cycle time units (TSC) (senders: tscrtps and tsclinuxps) tscskewd.c - it can be used to set the current TSC conversion constant and offset estimates or to continously estimate the TSC conversion constant using the standard SW clock (gettimeofday()), which is assumed to be synchronised to a primary NTP server ( default parameters assume max. 2ms offset in the NTP synch. SW clock ) VI. Some comments on Real-Time Networking ----------------------------------------- The RT-Net package implements a significantly cut down version of the standard Linux networking stack. It does not support TCP and its configuration is VERY different from the Linux networking. The network interface configured to work as a RT-Net interface becomes invisible for the Linux. If you are planning to use RT-Net on a single NIC machine you need first to bring down the Linux interface (eth0) and then to bring it up as an RTLinux interface (rtifconfig). As a result of this the host will be accessible ONLY FROM THE CONSOLE (no ssh, no telnet, no rlogin ...). RTNet routing is very simple, it is based on a IP to MAC address conversion table. When a packet is sent, the packet is delivered to the MAC address belonging to destinatinon IP address. There is NO DEFAULT routing. In this version of the RTNet new entries to the routing table can be added in the following ways: 1. Automatically - each time a packet arrives to the interface its soruce IP address is checked. If this address is not included in the routing table, then a new entry is created with the MAC address of the inteface from which the packet has arrived. THIS PROCESS IS A POTENTIAL SOURCE OF PROBLEMS. Many networks are configured to use different gateways for incoming and outgoing packets. As a result of this, the RTNet interface might not be able to correctly respond to ping messages. The problem can be avoided using manual configuration. 2. Manual configuration A. rtifconfig route solicit - this command can be used to force sending ARP messages - unfortunatelly this only works on the local LAN as there will be no responses to ARP requests to external hosts. In the case of external hosts either the automatic process has to be utilised or the following command: B. rtifconfig route add - this command creates a copy of an already existing routing entry and replaces the ip address of the original entry by in the new entry - to manually create a routing entry to an external host first use 'A' to add an entry to the gateway (if it doesn't exist yet) then use 'B' As the RT-Linux applications are implemented as kernel modules all the messages from them are directed to the system console. Each time a new entry is created in the routing table a message is sent to the console. Command 'B' also lists the existing entries on the console. VII. Installation and test ------------------------- It is assumed that RT-Linux-2.2 and RTNet-0.9.0fix is installed and that the following soft links are created in the /usr/src directory: linux - pointing to the linux kernel (most probably in /usr/src/rtlinux-2.2/linux) rtlinux - pointing to the rtlinux directory (most probably in /usr/src/rtlinux-2.2 ) rtnet - pointing to the RTNet-0.9.0fix (most probably in /usr/src/rtnet-0.9.0 ) It is also assumed that the host is running the rtlinux patched kernel. ( check: uname -a the version should be: 2.2.14-rtl2.2 ) Extract the rtps.tar.gz archive 1. tar -zxvf rtps.tar.gz Compile the package 2. cd rtps 3. make Test if it works :) Let's assume a test host with two NICs: - Linux interface: eth0 with ip address 10.0.0.104 - RT-Linux interface: eth1 with ip address 10.0.0.114 This host is going to be the Sender in our test. Let's assume that we are using host 10.0.0.8 as a Receiver which is located on the same LAN as the server. For more information on routing configaration for hosts on external networks see chapter V. Bring interface eth1 down (if it was configured as a Linux interface, you can check with ifconfig ) 4. ifconfig eth1 down Load rtlinux and rtnet modules: 6. ./startrt Configure it as a RT interface: 5. rtifconfig eth1 up 10.0.0.114 0.0.0.0 To check if it really works, ping 10.0.0.114 from the receiver machine, then check the console on the Sender. 6. tail /var/log/messages It should contain a message: 'rt_ip_route_input() adding route saddr=...' This shows that the packet has been received by eth1. If the replies to the ping message did not get back, most probably there is a problem with the routing configuration. For more details on this see chapter V. Now let's send a probe stream. Before starting the sender start tcpdump on the receiver host to be able to observe the arriving packets (eg. tcpdump -n udp port 7775) First we load in the RT-Linux sender application. This time we will use the sender using the standard sw clocks. 7. ./startrtps 10.0.0.8 7775 Then we start the Linux application delivering the packet sending schedule to the real-time application. 8. ./rtSend 1secx10.bin& (Note: if you don't use '&' then you will need to issue the next command from a different terminal window) The '1secx10.bin' example probe stream file defines a probe stream of 10 full size packets (1500 byte) with 1sec inter departure time. The tcpdump output on the receiver host should look like this: 15:40:52.739922 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 12 (DF) 15:40:53.940045 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 1472 (DF) 15:40:54.939927 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 1472 (DF) 15:40:55.939809 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 1472 (DF) 15:40:56.939711 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 1472 (DF) 15:40:57.939588 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 1472 (DF) 15:40:58.939481 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 1472 (DF) 15:40:59.939396 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 1472 (DF) 15:41:00.939261 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 1472 (DF) 15:41:01.939150 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 1472 (DF) 15:41:02.939040 eth0 < 10.0.0.114.1025 > 10.0.0.8.7775: udp 1472 (DF) An initial packet follwed by 10 packets 1 sec apart from each other. The reason for the inter arrival times in this tcpdump output being consistently below 1sec is the combined results of the skew of the sender and receiver sw clocks. To stop the sender application you need to remove the sender module. 9. ./stoprtps As a result of removing the rtps module, the 'sent.stscdt' was created, containing the TSC based timestamps of the packet sending times. (To be more precise, these timestamps are generated after the NIC was notified of a packet awaiting sending, and ordered to transfer it from the memory and transmit it.) The 'sent.stscdt' file needs to be converted to the '.dt' format for further analysis. This can be done using the 'stscdttodt' program. However to be able to perform the conversion the TSC conversion constant must be known (see chapter II.). An estimate of this constant can be obtained using the 'tscskewd' program. To get a quick conversion constant estimate: 10. ./tscskewd NTP > cc.txt& After 96 seconds read out the estimates either by using: 11. ./readtscskew Which reports: shm: count:0 time: 0 - if there is no estimate available yet or shm: count:6583241443 time: 11999346 The value can also be checked simply by reading the cc.txt file. The data stored in this file should be used to obtain long term averages of the conversion constant if required for higher accuracy. Note: the conversion constant is the count/time ratio, calculate the mean of the ratio to get a more accurate estimate Note: if your sw clock is not NTP synchronised the conversion constant will suffer the same skew as you local clock, which is most probabily in the range of 30-50 PPM After obtaining the components of the conversion constant the 'sent.stscdt' finally can be converted to '.dt' file. Use dt_ts.m Matlab function to load and process the '.dt' file for analysis in Matlab. 12. ./stscdttodt sent.stscdt 6583241443 11999346 The same conversion constant can be used for the tsc based sender and receiver programs. To start the TSC based sender use './startrtps tsc' To stop it use './stoprtps tsc' To use the RT-Linux receiver and receiver monitor programs use './startrtpr ' and './rttscreceiver filename.tscdt &'. To stop the receiver use './stoprtpr'. After finishing with the experiment use 'tscdtotodt filename.tscdt