Trace-Driven Scientific I/O Benchmarks
Introduction
Traces of scientific I/O workloads are being made available to enable computing-related
research. Examples include traces from
Sandia National Laboratories
and Los Alamos National Laboratory.
Useful statistics can be extracted from such traces. However, it is sometimes desirable
to run applications represented by the traces so as to evaluate the performance and other
behaviors of I/O and storage systems. For this purpose, we have created a trace player,
TracePlay/Control, written in C, to recreate some of the trace conditions in the form of
runnable benchmarks. Our trace player uses formatted traces that are derived from
original scientific I/O traces. Our player is less of a full-blown utility but more of
a benchmark using traces extracted from actual scientific applications. The benchmark
itself is largely just a shell or wrapper for a parsed trace.
Design
In order to successfully replay a trace, the file system context must be recreated.
Our trace player extracts directory and file names accessed throughout the trace and
recreates the hierarchy. Files are created with the maximum estimated size inferred from
I/O system calls (read, write, seek). Our trace player is capable of running in two modes:
- Sequential (using traceplay): Sequential mode is essentially batch mode for traces.
It can replay a single trace. If given a list of traces, it will replay those traces
one after another, i.e. sequentially.
- Parallel (using tracecontrol): In parallel mode, two or more traces are replayed
concurrently.
- unsynchronized: an input list of traces is replayed concurrently by parallel
processes through the fork system call. All processes are allowed to run at best
possible speed.
- synchronized: Since many scientific applications use concurrent processes
with some form of synchronization, we also support a synchronized replay mode.
Synchronized mode requires that some version of MPI be installed (MPICH-2 was used
in our system) as MPI calls in the traces are used to enforce synchronization.
Download
Download source: traceplayer_v0.9.zip
Traces below originated from those released by
Sandia National Laboratories
and Los Alamos National Laboratory.
We sanitized the original traces and converted them into the format suitable for our
trace player.
Sanitized and formatted Sandia traces:
sandia.alegra.2744.zip
sandia.cth.zip
sandia.s3d_fortIO_031709.zip
Sanitized and formatted LANL traces:
32PE_N-1_nonstrided_64K_MPI.zip
32PE_N-1_nonstrided_1024K_MPI.zip
Installation notes (also in the README)
- If you plan on using parallel synchronized mode, you must have some version of MPI
installed on the machine where you will be running. tracecontrol expects to find mpiexec in /usr/local/bin. If MPI is installed
in a different directory, the path in tracecontrol.c must be changed in line 476. MPI is not needed in other modes.
- Included is a makefile. To produce all executables:
make all
To produce an MPI synchronized version:
make mpi
To produce an unsynchronized version (no MPI needed):
make seq
- There are several compiler flags that can be used to turn on various debugging messages.
These are:
- -DDEBUG: turns on a lot of low-level debugging messages that describe where in the
reconstruction and execution of a trace that traceplay is in. Extremely verbose.
- -DMPI_DEBUG: turns on some debugging messages related to replay execution of MPI calls.
- -DRECON_DEBUG: turns on debugging messages during file system reconstruction.
- -DCREATION_DEBUG: turns on details of the file creation process.
- -DMPI_RUN: (applies only to traceplay.c) produces an MPI-version of the executable.
- -DLINES: (applies only to traceplay.c) periodically outputs line numbers as a trace is
replayed to show progress.
- traceplay.c must be compiled with mpicxx if using MPI.
Usage (also in README)
- To run 1 or more traces in sequential mode:
./traceplay target_mount_point trace_1 trace_2 ... trace_n
- To run 2 or more traces in parallel unsynchronized mode:
./tracecontrol target_mount_point trace_1 trace_2 ... trace_n
In this mode, multiple traceplay processes are forked
to run in parallel without synchronization between them.
- To run 2 or more traces in parallel synchronized mode:
./tracecontrol -mpi target_mount_point trace_1 trace_2 ... trace_n
In this mode, mpiexec is employed to run the processes in parallel.
MPI synchronization only applies to the LANL traces.
While the Sandia traces can run successfully with the -mpi flag,
they will essentially be running unsynchronized since those traces
do not contain any MPI calls.
All the above benchmark runs may take an optional -nofs and -drop flags.
If -nofs is set, the benchmark run will not try to reconstruct the filesystem in
the trace. Because file system reconstruction is time-consuming, it can be
helpful to utilize the -nofs flag on subsequent runs. If -drop is set, the file
cache will be sync'd and dropped before each replay. This feature is available
in Linux kernels 2.6.16 and newer. You must have root privileges to utilize the
-drop flag.
Supported research
-
Stan Park and Kai Shen,
"A Performance Evaluation of Scientific I/O Workloads on Flash-Based SSDs".
In Workshop on Interfaces and Architectures for Scientific Data Storage (IASDS'09),
New Orleans, LA, September 2009.
Abstract.
Postscript version.
PDF version.
Contact: Stan Park
email: park _at_ cs.rochester.edu
Support: This project is supported by the National Science Foundation under Grants No. 0937571 and No. 0621472.
Note: Any opinions, findings, and
conclusions or recommendations expressed in this material are those of the
author(s) and do not necessarily reflect the views of the National Science Foundation.