High-Fidelity Terascale Simulations of Turbulent

Combustion

Jacqueline H. Chen, Evatt R. Hawkes, Ramanan Sankaran, James

C. Sutherland, and Joseph C. Oefelein
Combustion Research Facility
Sandia National Laboratories
Livermore CA

Supported by
Division of Chemical Sciences, Geosciences, and Biosciences,
Office of Basic Energy Sciences, and DOE SciDAC

Computing: LBNL NERSC, ORNL CCS/NLCF, PNNL MSCF,

SNL HPCNP Infiniband test-bed,
SNL CRF BES Opteron cluster

Visualization: Kwan-Liu Ma and Hiroshi Akiba UC Davis

 Outline

1. DNS of turbulent combustion

challenges and opportunities

2. Sandia S3D DNS capability

terascale simulations on
Office of Science platforms

3. New combustion science to

advance predictive models:

3D simulations of turbulent jet

flames with detailed chemistry.

Layering Large Eddy Simulation Vorticity fields in DNS of a turbulent jet flame
and DNS approaches. (volume rendering by
Kwan-Liu Ma and Hiroshi Akiba)
Turbulent combustion is a grand challenge!

Stiffness : wide range of

length and time scales
turbulence
flame reaction zone
Chemical complexity
large number of species and
reactions (100s of species,
thousands of reactions)
Multi-Physics complexity
multiphase (liquid spray, gas
phase, soot, surface)
thermal radiation
acoustics ...
All these are tightly coupled

Diesel Engine Autoignition, Soot Incandescence

Chuck Mueller, Sandia National Laboratories
 Several decades of relevant scales

Typical range of spatial scales

Scale of combustor: 10 100 cm
Energy containing eddies: 1 10 cm
Continuum

O(4)
Range Small-scale mixing of eddies: 0.1 10 mm
Diffusive-scales, flame thickness: 10 100 m
O(4)
Molecular interactions, chemical reactions: 1 10 nm

Spatial and temporal dynamics inherently

coupled
All scales are relevant and must be resolved or
modeled

Terascale computing:
~3 decades in scales
(cold flow)
 Combustion CFD Approaches to tackle
different length scale ranges

ReynoldsAveraged NavierStokes (RANS)

fixed physical problem

Coarse meshes, full range of dynamic scales modeled

and resolution for
Increasing cost

Empirical closure, bulk approximation, current engineering CFD

Large Eddy Simulation (LES)

Energetic scales resolved, sub-grid scale dynamics modeled
combustion closure required, captures large-scale unsteadiness

Direct Numerical Simulation (DNS) uL

No sub-grid modeling required but limited on range of scales Re
Building-block configurations, research tool
 Role of Direct Numerical Simulation (DNS)

Oxidizer
A tool for fundamental studies of the micro-
physics of turbulent reacting flows Fuel
CH4
Physical insight into chemistry HO2

DNS turbulence interactions

Full access to time resolved
fields
CH3O O

A tool for the development and validation of

reduced model descriptions used in macro-
Engineering-level
scale simulations of engineering-level
Physical CFD systems
codes
DNS
Models (RANS and
future LES)
Piston Engines
 S3D MPP DNS capability at Sandia

S3D is a state-of-the-art DNS code developed with

13 years of BES sponsorship.
S3D code characteristics: S3D scales up to 1000s of
Solves compressible reacting Navier-Stokes processors and beyond?
F90/F77, MPI, domain decomposition.
Highly scalable and portable
8th order finite-difference spatial
4th order explicit RK integrator
hierarchy of molecular transport models
detailed chemistry
multi-physics (sprays, radiation and soot) from SciDAC
Terascale Simulation of Turbulent Combustion
70% parallel efficiency on 4096 processors on
NERSC; 90% parallel efficiency on 5120 CrayXT3
processors; 75% parallel efficiency on 1000
CrayX1E processors

Terascale computations => need scalar

optimization customized to architecture
CCS, NERSC consultants
 S3D performance improvements on Seaborg

Application Software Case Studies in FY05 for MICS (K. Roche)

Using Xprofiler, IPM and working with David Skinner:
8% improvement in performance by minimizing
conversion between dimensional and nondimensional
units for interfacing with Chemkin
interface to Mathematical Acceleration Subsystem
(MASS) library for exponential and logarithmic
functions resulted in 10% improvement. Another 5%
due to using vectorized exp function in VMASS
Tabulating Gibbs energy as a function of temperature
resulted in 8% savings
Remove excessive dynamic allocation calls resulted
in 12% savings.
Eliminate MPI derived data types for communicating
noncontiguous boundary data with neighbors 45% efficiency improvement
 Outline

1. DNS of turbulent combustion

challenges and opportunities

2. Sandia S3D DNS capability

terascale simulations on
Office of Science platforms

3. New combustion science to

advance predictive models:

3D simulations of turbulent jet

flames with detailed chemistry

Vorticity fields in DNS of a turbulent jet flame

Layering Large Eddy Simulation (volume rendering by
and DNS approaches Kwan-Liu Ma and Hiroshi Akiba)
FY05 INCITE award: Direct simulation of a 3D turbulent nonpremixed
CO/H2/air jet flame with detailed chemistry (350 million grids, 12 species)

Objectives:

z Study fine-grained coupling between

x
turbulent mixing and finite-rate
y chemical effects
Extinction-reignition dynamics
x Mixing time scales

Reynolds number effect

Preferential diffusion
Flow unsteadiness
INCITE run test case (1/2 resolved) Test assumptions in subgrid mixing
350 million grid points, Re=5000 and reaction models
Isoscalar surface of scalar dissipation rate (local mixing rate)
The effect of chemistry-turbulence interactions
in turbulent nonpremixed jet flames:
mixing of passive and reacting scalars
Evatt Hawkes, Ramanan Sankaran,
James Sutherland, and Jacqueline Chen

Computational Platforms: NERSC Seaborg, ORNL CCS Phoenix Cray

X1E, XT3 and PNNL MPP2
 Description of runs
- Temporally evolving non-premixed plane jet flames

A canonical flow with shear-driven turbulence

Air
Mixing,
Reaction
Fuel x
y
Mixing,
Reaction

Air
Heat release iso-contours
 DNS data-sets of turbulent nonpremixed
H2/CO flames (mixing, extinction/reignition)

Large computing allocations are enabling new science runs

INCITE at NERSC, capability computing at NLCF ORNL, ERCAP at PNNL, NERSC
Detailed H2/CO chemistry (17 d.o.f., Li et al. 2005)
Parameters selected to maximize Re INCITE calculation:
Currently running on Seaborg
Case A: Re 5000
Re 6000 350 million grid points
40 million grid points ~4096 Seaborg processors
3 million hours total
480 processors on PNNL MPP2
~ 9TB raw data
~ 1 TB data

Case B: Two additional calculations,

parametric study in Re
Re 8000
Re 2500, 150 million grid points
100 million grid points
Re 10,000, 500 million grid points
1728 processors on Seaborg
192/240 processors on ORNL CCS Cray X1/E
~ 2.5 TB data
Community data sets

New opportunities for numerical benchmarks highly resolved LES and DNS

TNF workshop (1996-2004): International Collaboration of Experimental and

Computational Researchers
Framework for detailed comparison of measured and modeled results
(http://www.ca.sandia.gov/TNF/abstract.html)
Non-premixed combustion concepts

Mixture fraction Z: the amount of fluid from the fuel stream in the
mixture

Z is a conserved (passive) scalar (no reactive source term)

Scalar dissipation, a measure of local molecular mixing rate:

2 D Z Z
Example development of the jet:
40 million grid PNNL run
Left: Vorticity. Right: Simultaneous volume rendering of mixture
fraction, scalar dissipation and OH radical.
(Rendering by Hiroshi Akiba and Kwan-Liu Ma, UC Davis)
100 million grid run on
NERSC Seaborg and ORNL Cray X1: Re = 8000

Scalar Dissipation Vorticity

2 D Z Z u
100 million grid run on
NERSC Seaborg and ORNL Cray X1: Re = 8000

HO2 dissipation OH dissipation

HO2 2 DHO2 YHO2 YHO2 OH 2 DOH YOH YOH
Mixing timescales

Models for molecular mixing are required in the PDF

approach to turbulent combustion (Pope 1985), a sub-grid
model used in RANS and LES approaches.

TNF workshop CFD predictions are dependent on mixing

timescale choice.

Models assume that scalar mixing timescales are identical

for all scalars and determined by the turbulence timescale.
scalars with different diffusivities?
reactive scalars?
Definitions

k
Mechanical time-scale: u

' 2
Scalar time-scale:

2 D
Time-scale ratio: u
r

r is assumed to be order unity in most models
r is assumed to be the same for all scalars
Mixture fraction to mechanical timescale
ratio

Confirmation that mixture fraction to mechanical time scale ratio is

order unity.
Average value about 1.5, similar to values reported by experiments,
simple chemistry DNS, and used successfully in models.
Timescale Ratio rZ
 Effect of diffusivity

Increasing
diffusivity

r
Smaller, more highly
diffusive species do
have faster mixing
timescales
 Finite-rate chemistry effects on mixing

r
HO2 and H2O2 have faster
mixing times in the middle
of the simulation, while
OH and O are lower

Diffusivity trend does not

appear to hold for HO2 and
H2O2 versus O and OH.

What is going on?

Radical production and destruction in high
dissipation regions
Color scale: mass fraction (blue is low; red is high)
White iso-contours: scalar dissipation rate

HO2 OH

OH is consumed while HO2 is produced in high dissipation

regions
Dissipation of passive and reactive scalars

Blue: Z, Green: OH, Red: HO2

Dissipation fields of Z and HO2 are

co-incident and aligned with principal
strain directions

OH dissipation occurs elsewhere,

more in the centre of the jet

OH is reduced in the high dissipation

regions, leading to longer mixing
times, opposite for HO2

At later times, mixing rates relax, OH

returns and HO2 decreases
 100 million grid run on Seaborg / Cray X1

We need to establish
any Re dependence

Need parametrics at
larger Re

Run in progress
but seems to be
showing same effects
Conclusions - mixing timescales

New finding: detailed transport and chemistry effects can

alter the observed mixing timescales

Models may need to incorporate these effects

a poor mixing model could lead to incorrectly predicting a stable
flame when actually extinction occurs

This type of information cannot be determined any other

way at present
ambiguities in a-posteriori model tests
too difficult to measure
need 3D and detailed chemistry to see this
Knowledge Discovery From Terascale Datasets

Challenge: Need interactive

knowledge discovery software for
terascale high-fidelity scientific data
sets where computing pipeline is
distributed geographically.
Solution: Intelligent visualization
pipeline
Simultaneous cognitive
visualization
Intelligent feature
extraction/tracking
INCITE Award: DNS of a 3D Scalable transparent data sharing
turbulent flame (9 TB raw data - 350 and parallel I/O across platforms
million grids, 17 variables, 100,000
steps)
Advanced post-processing for extinction/re-ignition

Extinction viewed as radical or

reaction-rate hole in a mixture-
fraction isosurface

Stems from fast chemistry flame-

sheet flamelet models of
turbulent combustion

Holes can be expanding or

contracting

Speed of the edge can be

monitored need algorithm for
edge detection
Advanced post-processing for extinction/re-ignition

Holes could be tracked to determine any history effects

need parallel feature tracking
 Outline

1. DNS of turbulent combustion

challenges and opportunities

2. Sandia S3D DNS capability

terascale simulations on
Office of Science platforms

3. New combustion science to

advance predictive models:

2D DNS of HCCI combustion

3D simulations of turbulent jet flames

with detailed chemistry
Vorticity fields in DNS of a turbulent jet flame
Layering Large Eddy Simulation and (volume rendering by
DNS approaches Kwan-Liu Ma and Hiroshi Akiba)
Layering of LES and DNS approaches
(the future...)

Joseph C. Oefelein,
Evatt R. Hawkes, Jacqueline H. Chen
RANS-LES-DNS at High Re
Swirl-Stabilized Premixed Combustion
Azimuthal Velocity Component

RANS LES DNS

LES gets lots more

than RANS

High Re is too
expensive for DNS
(at present)
can afford 1000s now
useful but
would like 100000s
and more
Why LES?

LES treats the large

scales directly

Large scales are

geometry
dependent

LES can couple

directly with high Re
experiments

But LES needs

sub-grid models
Layering LES and DNS

Better BCs, ICs

Identify modelling
questions
Identify relevant
parameter space

Moderate Re:
High Re Canonical problems
LES DNS
Experiments Cleverly designed
experiments

Sub-grid models
Applications
 Multiphysics capabilities and
algorithmic framework Oefelein

Theoretical framework (LES DNS)

Fully-coupled conservation equations
Compressible, chemically reacting system
Real-fluid EOS, thermodynamics, transport
Detailed chemistry, multiphase flow, sprays
Dynamic subgrid-scale modeling

Numerical framework
Dual-time, all-Mach-number formulation
Generalized preconditioning methodology
Complex geometry, generalized coordinates
Massively-parallel (MPI), highly-scalable
Ported to all major platforms

Over 10 years development Fine-grain scalability (NERSC IBM SP Seaborg)

Grid size fixed, number of processors increased
 Magnitude of vorticity (normalized)

DNS LES
(all scales) (large scales)
40 million grid, 120,000 hours 0.6 million grid, 600 hours
Summary

S3D is a state-of-the-art DNS capability for turbulent combustion

simulations that scales to thousands of processors and is ported to
all Office of Science platforms.

DOE supercomputing facilities are enabling new combustion

science.

3D DNS of detailed finite-rate chemistry effects in turbulent jets

provides new insights and data for combustion modeling:
mixing of reactive scalars can be very different from conserved
scalars

Layering LES and DNS approaches will open new avenues of

exploration with great potential for the future
100 million grid run on
NERSC Seaborg and ORNL Cray X1: Re = 8000

HO2 dissipation OH dissipation

S3D performance on different architectures