Test analysis & comparison
Companioning paper written by Frank Brüseke, Henning Wachsmuth, Steffen Becker, Gregor Engels
Licenses and Copyright
- The scripts and models on this website are made available under the regulations of the GNU Lesser General Public License LGPL
- The data sets, the resulting diagrams and tables, and also this website are made available under the regulations
of the Creative Commons Attribution-ShareAlike 3.0 Unported License
Prerequisites
- Ensure that the R "bin"-directory is on your PATH environment variable
- Be sure you know the path to your dedicated folder "<dataPath>", where you keep all example data and scripts
- Download the All-in-one script zip-file: [Zip]
- Unpack the Zip to <dataPath>
- Download the raw data series file to <dataPath>: [Zip]
- Unpack the test data series to <dataPath>
- Download the PCM (aka Palladio) data series: [Zip]
- Unpack the PCM data series to <dataPath>
- Download the conversion factor configuration file to <dataPath>: [File]
Steps
Hints:
Most of the steps are done at the command line, the python scripts usually support parameters and the "--help" parameter will show sensible help messages.
If you cannot start python scripts directly prefix "python" or "python.exe" to the command line (assuming the python executable is on your PATH)
- Open a command line and "cd" to <dataPath>
- Convert test data series to performance blame analysis format:
- Short cut to step 4:
download the converted data series file to <dataPath>: [File] - On the command line use the command:
java -jar ./KiekerToPba/KiekerToPba.jar -i . - This will produce the file "stack-traced.out.txt" with all measurements in the correct format. The conversion script constructs stack traces from the Kieker measurements.
- This will also produce the file "stack-traced.map.txt"
which contains the form for the PCM to implementation mapping. It
contains all the different stack traces observed in the data file along
with their compressed name. In the third column the system architect has to
fill in the matching PCM simulation data series (as relative path to the file containing the PCM simulation data series). In this example you do not need to do this as we will work with a pre-made mapping file called "stack-traced.map.txt".
- Download the PCM to Implementation mapping file to <dataPath>: [File]
This overwrites the file "stack-traced.out.txt". The file you download contains a complete mapping with the path of each Palladio measurement file. In contrast the file just generated lacked this information. - Configure conversion factors:
- The conversion factors are configured in the file "factors.conf". This is a line-based configuration file. The first line stands for the conversion factor for the test data series, while the second line stands for the factor for the PCM data series. In this example the configuration looks like this:
0.001*0.001
The second line is empty, which means that there is no conversion needed for the PCM data series. The first line gives an R formula that is prefixed as a factor to the line where the data series is read. I.e. the syntax of the factor formula must be correct. - Start the analysis script with the following command:
compareTestAndSimu.py -m stack-traced.map.txt -t stack-traced.out.txt -o analysis01 -f factors.conf - If all goes well this command produces a lot of files:
- The performance report: analysis01.RR.pdf
- The extended performance report: analysis01.R.pdf
- A series of single box plots (for the nice figures in the paper)
- The flame graph: analysis01.flame.csv2.svg
- For troubleshooting refer to the command line output and the file: analysis01.R.out.txt
- All other file are intermediate data or script files
