Time step from the evolution of temporal mixing layer from initial to vortex breakdown. AVF/Leslie simulation performed using up to 131,072 cores on DOE's TITAN supercomputer. In situ processing using VisIt software delivered the images related plots and XDB files for the researchers. Time step 150,000 out of 200,000.
This summer I presented a talk called In Situ Production of Extract Databases for Visualization to the Workshop on In Situ Visualization at the ISC Conference held in Frankfurt, Germany. In situ takes workflows that have been created post hoc and executes them directly on simulation data in memory while the simulation is running. The workshop was attended by visualization experts mainly from Europe and the US and talks focused on using in situ software to address the challenges of being able to save sufficient simulation data on supercomputers.
Hank Childs from the University of Oregon, a prominent visualization expert, gave a keynote address emphasizing the importance of in situ against the backdrop of upcoming exascale machines with their diminishing memory per core and lower relative I/O bandwidth compared to today's machines. Jens Henrik Goebbert from Aachen University presented an abstraction library that simplifies in situ integration with multiple in situ infrastructures, including Libsim. Roberto Sisneros from NCSA presented work on a parameter study highlighting the importance of providing good default application settings and showed that performance for VisIt's streamline plot could be enhanced by simply improving the default settings.
My talk summarized Intelligent Light's in situ efforts with VisIt, Libsim and our extract database files (XDB). Specifically, we instrument a simulation for in situ using Libsim, which brings VisIt's capabilities into the simulation. We developed a library that efficiently writes the FieldView XDB files and added it as an export option to VisIt. The simulation uses VisIt to create surface geometry extracts, typically without making any copies of simulation data, and exports the extract as a FieldView XDB file. Tight coupling of simulation to visualization and analysis provides opportunities to perform data reductions that result in smaller, concentrated, more useful results being written out more frequently, avoiding the costs of writing and later reading large volume datasets.
I presented results from running the AVF-LESLIE combustion code on the Titan machine at Oak Ridge National Laboratory using an in situ rendering workflow and our extract database workflow. For the rendering workflow we were able to run the code up to 131K cores and render images of slices and isosurfaces from the simulation every 5 time steps to produce a visualization of a turbulent mixing layer of 2 fluids. In another of our experiments, we extracted surface-based results, saving the geometry plus field data to our XDB format for later post-processing within FieldView. We saved XDB extracts from every 5th time step, taking around 2% of the simulation runtime. Each XDB file was over 200x smaller than the corresponding volume data file. We wrote 20 XDB files for every volume output file and the combined size of 20 XDB files was still 10x smaller than a single volume output file.
This work is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research under Award Number DE-SC0012449.