2009 DoD High Performance Computing Modernization Program Users Group Conference

Towards Prediction of 3-D Separated Flows Using an Unstructured LES Code

Peter Chang, Minyee Jiang, and Dory Lummer US Naval Surface Warfare Center, Carderock Division (NSWCCD), West Bethesda, MD {peter.chang,minyee.jiang,dory.lummer}@navy.mil Abstract
In this paper, we document large eddy simulations (LES) performed on a sphere and the Advanced Swimmer Delivery System (ASDS), a small blunt-ended submarine. The objective of this work is to develop a methodology for being able to compute the occurrence and strength of stern separation in order to accurately predict drag, maneuvering and structural loads, and acoustic signatures. We are using an energy-conserving large eddy simulations (LES) code called MPCUGLES that runs with hybrid structured-unstructured meshes. We document LES of flow over a sphere at Reynolds numbers 10,000 to 1.14 million with excellent comparison to experimental data. We also document a preliminary effort for LES of flow over ASDS. Even the simplest of these computations is very CPU-intensive necessitating the large amounts of CPU time available through the HPCMP Challenge Project C3U.

Krishnan Mahesh Department of Aerospace Engineering, University of Minnesota, Minneapolis, MN Mahesh@aem.edu

adverse pressure gradient regions that makes the prediction of the occurrence of separation suspect. Except for the cases of highly separated flows (e.g., flow about a square cylinder) RANS cannot resolve the complex vortex structures in separated flows because of the dissipative nature of the turbulence closure and discretizations. A promising alternative to RANS is thought to be DES, whereby RANS methodologies are used to simulate the attached flows on the body and LES is used in the separated regions. The drawback with this method is that there is a grey region where the RANS modeling ends and the LES region commences, the flow undergoes a relaminarization where the turbulence is neither modeled nor has had time to develop physicallycorrect flow turbulence flow structures. This relaminarization can be the cause of early and stronger flow separations (Chang, 2004). LES, in theory, has the most potential for correctly predicting separation, because it resolves the turbulent flows in the attached flow region as well as in the separated flow regions and therefore, in theory, should be able to predict the occurrence and strength of separation. After the flow has separated, LES has the numerical accuracy to predict the turbulence scales important for acoustic and structural problems. In reality, however, most practical separated flows have high Reynolds numbers meaning that resolution of physicallycorrect turbulence structures in the incident, attached boundary layers requires extremely large computational resources. The Advanced Swimmer Delivery System (ASDS) is an 80-foot long submersible that rides atop attack submarines (Figure 1). The pylon support structures and the mating trunk, as well as the ASDS body itself, are partially streamlined bodies that are the types of geometry for which the prediction of the occurrence of separation is very difficult. Extensive experiments have been performed on ASDS in NSWCCDs 36 Variable Pressure Water Tunnel (VPWT) and Large Cavitation Channel (LCC) in order to measure forces on the afterbody, appendages, and propulsor.

1. Introduction
Separated flows about submerged bodies are the cause of unwanted drag, noise and structural excitation. In order to maximize payload while minimizing body length and cost, ship designers must balance the need for internal volume with the need to avoid flow separation from sharply converging stern lines. The ability to predict the occurrence of stern, or trailing edge separation on bodies with continuously converging lines (as opposed to sharp transom-type lines) is one of the great challenges for ship designers, experimentalists, and computationalists, alike. To date, computational methodologies, including Reynolds-Averaged NavierStokes (RANS), detached eddy simulations (DES), as well as large eddy simulations (LES), are unable to reliably predict the occurrence of separation and the comcomitant strength of vortex shedding that provides the forcing functions for acoustic power and structural excitation on objects in the bodies wakes. RANS methods suffer from turbulence modeling deficiencies in

978-0-7695-3946-1 2010 U.S. Government Work Not Protected by U.S. Copyright DOI 10.1109/HPCMP-UGC.2009.23

127

The unstructured LES code, MPCUGLES, has been used extensively for predicting propeller crashback (Vysohlid and Mahesh, 2006, Chang et al., 2008a,b). In crashback, the propeller is reversed while the boat is moving forward, so that the trailing edges of the propeller blades become the leading edges. Since the dominant force-producing physics are flow separations that occur on this new leading edge, the attached flow over the rest of the blade is not of critical importance. For flows over ASDS, the upstream flow IS attached and separated flow is at the stern, thus placing a much more critical emphasis on the accurate simulation of the upstream flows. Since this is a new class of problems for our LES capability, we are taking a step-by-step approach by first simulating the flow about a sphere, then moving on to the ASDS, working first on the ASDS body itself, and finally, the ASDS as mounted on a submarine. The sphere has many of the same computational issues as the ASDS except that it has a much simpler geometry, which makes investigation of the onset of separation and strength of vortex shedding more expedient. In comparison to the flow over a cylinder, the sphere has a relatively weak adverse pressure gradient (Achenbach, 1972) with three-dimensional separation, much like what would be expected for ASDS. The flow over the sphere is dependent upon the Reynolds numbers1: for 1,000<Re<375,000 the flow is subcritical. On the sphere and in the near wake the flow is laminar with the downstream wake becoming turbulent. For Re>375,000, the flow is super-critical. The flow on the sphere undergoes laminar to turbulent transition on the sphere itself and has a turbulent wake. The so-called drag-crisis occurs at Re=375,000, where the switch between sub- and super-critical flows causes the drag to drop by a factor of three. Achenbach (1974) found that at sub-critical Reynolds numbers strong periodic shedding is observed, whereas for super-critical Reynolds numbers periodic vortex shedding is not present. Previous studies that used turbulence simulations were performed by Constantinescu and Squires (2004) who performed structured grid DES calculations on a sphere in sub- and super-critical regimes. Kim (2004) performed LES over a wide range of Reynolds numbers using ANSYS-Fluent LES with unstructured grids. In this paper, we first review the mathematical formulation for MPCUGLES showing why it may be an ideal code for large separated flow problems; we show results for canonical flow over a sphere at a range of Reynolds numbers; we then discuss the extensive testing that has been performed with MPCUGLES on the ASDS geometry. Finally, we make conclusions.

2. Numerical Methodology
The incompressible LES equations are obtained by spatially filtering the Navier-Stokes equations assuming that the filtering operation commutes with the spatial and temporal derivatives. The dynamic SGS model of Germano et al., (1991) and modified by Lilly (1992) is used to model the SGS stresses. The filtered momentum equations are solved using a numerical method developed by Mahesh et al., (2004) for incompressible flows on unstructured grids. The finite volume algorithm, derived to be robust with minimal numerical dissipation, stores the Cartesian velocities and pressures at the control volume (CV) centroids with the face-normal velocities stored independently at the centroids of the faces. A predictor-corrector approach is used where the predicted velocities at the CV centroids are first obtained then interpolated to obtain the face-normal velocities. The predicted face-normal velocity is projected such that continuity is discretely satisfied. This yields a Poisson equation for pressure, which is solved by algebraic multigrid (AMG) (Henson, 2002) or conjugate gradient (CG) approaches. The pressure field is used to update the Cartesian CV velocities using a least-squares formulation that is essential for obtaining solutions on highly skewed grids. Time advancement is performed using an implicit Crank-Nicholson scheme. Figure 2, from Chang et al., (2008), compare the speedups for solving large problems on massively parallel compute clusters using CG and AMG methods showing that for a 13 million CV propeller crashback problem, using the CG Poisson solver, the code has near linear speed up for up to 400 processors.

3. Results and Discussion

3.1. Flow About Sphere
The ASDS hull form will have very weak shedding and therefore, the flow over a sphere, without a strong adverse pressure gradient and shedding is a good problem to develop a methodology for separated flows for ASDS. We use triangles on the sphere surface, prism layers extruded off the body, tetrahedra outside of this and finally, hexahedra or prisms extending to the upstream and downstream boundaries. While the sphere could have easily been meshed with a structured grid, this hybrid structured-unstructured mesh will allow us to more easily handle arbitrary, complex structures that are present, in say, the ASDS mounted on a submarine. No slip boundary conditions are specified on the sphere surface, constant velocity boundary condition are specified at the inflow and lateral boundaries and a convective condition is set at the downstream boundary.

1 For the sphere, the Reynolds is number is based on freestream velocity, sphere diameter, and kinematic viscosity.

128

Six grids have been developed to ascertain the effects of spacing on the wall, normal to the wall and in the wake. In this paper, we just report on grid sphere-2 which has a uniform triangular grid on the surface with dimension 0.0266D and wall normal spacing 0.0011D. The wall layers are shown in Figure 3 and in Figure 4 a fine grid region in the wake that supports vortex shedding can be seen. Sphere-2 has approximately 1.6 million CV. The viscous wall normal spacing is 0.44 wall units at Re=10,000 and 48 wall units at Re=1.14 million, so the higher Reynolds number is a test of what happens when the wall layers are under-resolved. Computations have been performed on ARL Stryker and Humvee on 64 processors. With a non-dimensional time step size tU/D=0.005, 100,000 time steps are needed for simulations where the flow progresses 500D. Approximately 100,000 CPU hours on ARL Stryker are necessary for each Reynolds number case. Long time series (500D/U) are necessary to characterize the low frequency wake shedding behavior. Figure 5 and Figure 6 show contours of u (total streamwise velocity) on the sphere centerplane and contours of Cp (pressure normalized by density and freestream velocity) on the sphere surface for Re=10,000 and 1.14 million, respectively. The figures show that, as expected the sub-critical Reynolds number flow separates earlier than for the super-critical flow because a turbulent flow (the super-critical case) is more stable since it has more momentum near the wall. The azimuthal angle of separation (s) for the four Reynolds numbers run for sphere-2, listed in Table 1, show that the two sub-critical cases (Re=10,000 and 42,000) separate slightly earlier than the experiments. For the super-critical cases, the separation point moves aft around the sphere. For Re=420,000, the flow separates somewhat earlier than the experiment while 1.14M compares exactly with the experimental value.
Table 1. Separation points for sphere-2 Re
10,000 42,000 420,000 1.14E+006
o s ( LES )

ANSYS Fluent LES and Constantinescu and Squires (2004) who used a DES approach. Figure 8 and Figure 9 are the azimuthally and timeaveraged pressure distributions over the sphere for the sub- and super-critical cases, respectively. For the subcritical case, the minimum pressure at about 70 is only about Cp=0.5, with a pressure recovery (the so-called base pressure) only back up to about Cp=0.3. The small pressure recovery accounts for the relatively large CD for the sub-critical cases. The MPCUGLES results have a slightly higher base pressure than Constantinescu and Squires (2004) which could help to explain why MPCUGLES has a slightly lower drag coefficient. At the super-critical Reynolds numbers, Figure 9, the pressure minimum is much lower, around Cp=1.15, with a base pressure of Cp=0.1. The recovery to slightly higher pressure than for the sub-critical cases explains why the super-critical cases have smaller CD. A potential flow solution exhibits DAlemberts paradox where the pressure is symmetric about 90 resulting in zero drag. Overall, the results for sphere-2 compare favorably to published results and have given us a good idea of the grid densities needed for simulation of weakly separated flows. Additional mesh refinements have been performed and results from them will be published in the future. We have performed extensive spectral analyses, which compare favorably to published data and bring to light differences between our energy-conserving LES scheme and the DES used by Constantinescu and Squires (2004). The spectral results for sub-critical cases have a strong peak at St=0.17 indicating strongly periodic shedding and no peak for super-critical cases, consistent with Achenbach (1974).

3.2. Flow About ASDS

In this preliminary work, we study the flow about ASDS placed in the center of a cylindrical tunnel. This approach allows us to focus just on the gridding issues for the ASDS body. We are studying optimal methodologies for generating prism layers from the triangular surface meshes on the body, especially in the vicinity of hullappendage junctures, Figure 10. This Challenge Project has allowed us to quickly test numerous variations. The grid topology that we have been working with is triangular surface mesh on the ASDS body, prism layers extruded from the body, transitioning into tetrahedral elements and a hexahedral cell region around this, extending to lateral and upstream and downstream boundaries. A no-slip boundary is applied on the body, constant velocities applied at the inflow and lateral boundaries and a convective boundary condition at the outlet. The grid sizes range from 2 to 16 million control volumes.

so ( Expt )
82

74 72 104 115

114118 114118

The drag on the sphere as a function of Reynolds number is shown in Figure 7. In the sub-critical range (Re=1,000 to 375,000) MPCUGLES is slightly low in comparison with the experimental data of Schlichting (1968), with the drag values dropping dramatically for higher Re, correctly predicting the drag crisis. The plot also shows the predictions of Kim (2004) who used

129

In this paper, we report the results of only Grid-5b, a 13 million control volume bare hull ASDS configuration, located in the center of a cylindrical domain with approximately the same radius as the LCC wall locations. The simulations of Grid-5b were performed at Reynolds number 2.12 million, based on maximum beam. The grid was split into 256 partitions and run on ARL Humvee for which one time step takes 2.7 wall clock seconds consuming 687 CPU seconds. The flow has been run for 40 body lengths (19 seconds of physical time data) consuming 122,000 CPU hours. Note that the ASDS experiments were run for 320 seconds in order to obtain a well-resolved frequency spectra. Such a simulation would consume 2.5 million CPU hours on Humvee. Figure 11, a center plane cut with contours and vectors of the fluctuating (total mean) velocities in the stern region, shows the existence of large vortices indicating a weakly separated flow. Figure 12 shows isosurfaces of 2, a variable that shows regions of coherent turbulent activity (Jeong and Hussain). One of the challenges that we have encountered with the ASDS is the existence of numerical dispersion at the tet/hex boundaries. However, as can be seen in the 2 isosurfaces, the turbulence generated by dispersion away from the hull is small in comparison with that near the wall. Thus, it may be possible to conclude that the nearwall flows, and thus, separation, are independent of the dispersion in the outer flow.

was made available at the DoD Supercomputing Resource Centers at Army Research Laboratory, Linux Clusters, Stryker, Humvee and Sketch. We would like to thank Joseph F. Slomski and Joseph J. Gorski for their advice and encouragement. We are grateful to George Constantinescu for providing us with sphere data. We would also like to thank Mark Motsko and Brian Simmonds at ARL for helping to maintain the V2D graphics streaming hardware.

References
Achenbach, E., Experiments on the flow past spheres at very high Reynolds numbers. Journal of Fluid Mechanics, 54, pp. 565575, 1972. Achenbach, E., Vortex shedding from spheres. Journal of Fluid Mechanics, 62, pp. 209221, 1974. Chang, P.A. and J.J. Gorski, Exploratory Hybrid RANS/LES of a NACA 0012 Airfoil. NSWCCD 50TR2004/019, Dec. 2004. Chang, P.A., M.P. Ebert, K. Mahesh and J. Shipman, Prediction of high amplitude forces during propeller crashback. DoD High Performance Computing Modernization Program 2008 Users Group Conference, Seattle, WA, 2008a. Chang, P.A., M.P. Ebert, S. Jessup, K. Mahesh, H. Jang, Y.L. Young, Z. Liu, and M. Shearer, Propeller Forces and Structural Response due to Crashback. 27th Symposium on Naval Hydrodynamics, Seoul, Korea, 2008b. Germano, M., U. Piomelli, P. Moin, and W.H. Cabot, A dynamic subgrid-scale eddy viscosity model. Physics of Fluids A Fluid Dynamics, 3, 7, pp. 17601765, 1991. Henson, V.E. and U.M. Yang, BoomerAMG: a Parallel Algebraic Multigrid Solver and Preconditioner. Applied Numerical Mathematics, 41, 2002. Jeong, J. and F. Hussain, On the identification of a vortex. Journal of Fluid Mechanics, 285, pp. 6994, 1995. Kim, S.-E., Large Eddy Simulation Using Unstructured Meshes and Dynamic SubgridScale Turbulence Models. 34th AIAA Fluid Dynamics Conference and Exhibit, Portland, OR, 2004. Lilly, D.K., A proposed modification of the Germano subgridscale closure model. Physics of Fluids A, 43, pp. 633635, 1992. Mahesh, K., G.S. Constantinescu, and P. Moin, A numerical method for large-eddy simulation in complex geometries. Journal of Computational Physics, 197, pp. 215240, 2004. Schlichting, H., Boundary-Layer Theory, New York, McGraw Hill, 1968. Vysohlid, M. and K. Mahesh, Large Eddy Simulation of Crashback in Marine Propellers. 44th AIAA Aerospace Sciences Meeting and Exhibit, Reno, NV, 2006.

4. Conclusions
In this paper, we give a brief overview of the effort to develop an unstructured grid LES tool that can efficiently and accurately predict the occurrence of stern separation. Using HPCMP resources with Challenge Project C3U we have been able to run numerous large LES to develop optimal gridding strategies. Six sphere grids at four Reynolds numbers have been run. Results show that the LES can predict forces, pressures and separation points over a range of sub- and super-critical Reynolds numbers. Preliminary runs have been completed on ASDS, a small, blunt-ended submarine. We are slowly building up the grid complexity and in this paper only report results for a bare hull body. Additional results including the ASDS mounted to a submarine, allowing for comparison to experimental data will be forthcoming.

Acknowledgments
This work was funded by the Office of Naval Research (ONR) (Ron Joslin and David Hess, program managers). The DoD HPC Modernization Program supported this project by supplying supercomputer time under the Computing Challenge Project C3U. This time

130

Figure 5. Contours of axial velocity for sphere-2 at Re=10,000. Sphere surface shows pressure contours.

Figure 1. ASDS in transit aboard attack submarine

Figure 6. Contours of axial velocity for sphere-2 at Re=1.14M. Sphere surface shows pressure contours.

Figure 2. Speedup study for MPCUGLES

Figure 7. Drag coefficients for sphere

1.5

Pressure Distributions for Sub Crit

1

Figure 3. Close-up of sphere grid

Cp

0.5

M-Re=10k M Re=42k CS Re=10k

-0.5

-1

-1.5

30

60

(degrees)

90

120

150

180

Figure 8. Pressure distribution over sphere for sub-critical cases

Figure 4. Centerplane cut of sphere grid showing outer tetrahedral and hexagonal grids

131

1.5

Pressure Distributions for Super Crit

1 0.5

M Re=420k M Re=1.1M CS Re=1.14M Ideal

Cp

-0.5

-1

-1.5

30

60

(degrees)

90

120

150

180

Figure 9. Pressure distribution over sphere for super-critical cases

Figure 11. Contours and vectors of fluctuating velocity over stern of bare hull ASDS (Grid-5b)

Figure 10. Typical surface mesh and extruded prisms in ASDS appendage-hull juncture

Figure 12. Iso-surfaces of 2 over stern of bare hull ASDS (Grid-5b) colored by axial velocity

132