Intelligent Light and FieldView

Visualization Engineer

ISAV 2016: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization

In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization

This workshop is held in conjunction with SC16 and is chaired by Wes Bethel, Senior Computer Scientist at Lawrence Berkeley National Laboratory.  Intelligent Light's Earl Duque, PhD. serves on the organizing committee and both he and I are members of the program committee for the full day event.  The workshop features a series of short papers, presentations and discussions.  

Workshop Theme: 

The considerable interest in the HPC community regarding in situ analysis and visualization is due to several factors. First is an I/O cost savings, where data is analyzed/visualized while being generated without first storing to a file system. Second, the potential for increased accuracy, where fine temporal sampling of transient analysis might expose some complex behavior missed in coarse temporal sampling. Third, the ability to use all available resources, CPUs and accelerators, in the computation of analysis products.

The workshop brings together researchers, developers and practitioners from industry, academia, and government laboratories developing, applying, and deploying in situ methods in extreme scale, high performance computing. The goal is to present research findings, lessons learned, and insights related to developing and applying in situ methods and infrastructure across a range of science and engineering applications in HPC environments; to discuss topics like opportunities presented by new architectures, existing infrastructure needs, requirements, and gaps, and experiences to foster and enable in situ analysis and visualization; to serve as a "center of gravity" for researchers, practitioners, and users/consumers of in situ methods and infrastructure in the HPC space.

I am a co-author on this paper to be presented during the workshop, which describes a common interface that can be used to instrument simulations with multiple in situ infrastructures, including Libsim:

The SENSEI Generic In Situ Interface
(Utkarsh Ayachit, Brad Whitlock, Matthew Wolf, Burlen Loring, Berk Geveci, David Lonie and E. Wes Bethel)

The full workshop agenda is available from the ISAV2016 webpage.

Attending SC16? come see us in booth #4421 or Request a meeting with our team​.

ISC Workshop on In Situ Visualization

Time step from the evolution of temporal mixing layer from initial to vortex breakdown. AVF/Leslie simulation performed using up to 131,072 cores on DOE's TITAN supercomputer. In situ processing using VisIt software delivered the images related plots and XDB files for the researchers. Time step 150,000 out of 200,000.

This summer I presented a talk called In Situ Production of Extract Databases for Visualization to the Workshop on In Situ Visualization at the ISC Conference held in Frankfurt, Germany. In situ takes workflows that have been created post hoc and executes them directly on simulation data in memory while the simulation is running. The workshop was attended by visualization experts mainly from Europe and the US and talks focused on using in situ software to address the challenges of being able to save sufficient simulation data on supercomputers.

Hank Childs from the University of Oregon, a prominent visualization expert, gave a keynote address emphasizing the importance of in situ against the backdrop of upcoming exascale machines with their diminishing memory per core and lower relative I/O bandwidth compared to today's machines. Jens Henrik Goebbert from Aachen University presented an abstraction library that simplifies in situ integration with multiple in situ infrastructures, including Libsim. Roberto Sisneros from NCSA presented work on a parameter study highlighting the importance of providing good default application settings and showed that performance for VisIt's streamline plot could be enhanced by simply improving the default settings.

My talk summarized Intelligent Light's in situ efforts with VisIt, Libsim and our extract database files (XDB). Specifically, we instrument a simulation for in situ using Libsim, which brings VisIt's capabilities into the simulation. We developed a library that efficiently writes the FieldView XDB files and added it as an export option to VisIt. The simulation uses VisIt to create surface geometry extracts, typically without making any copies of simulation data, and exports the extract as a FieldView XDB file. Tight coupling of simulation to visualization and analysis provides opportunities to perform data reductions that result in smaller, concentrated, more useful results being written out more frequently, avoiding the costs of writing and later reading large volume datasets.

​I presented results from running the AVF-LESLIE combustion code on the Titan machine at Oak Ridge National Laboratory using an in situ rendering workflow and our extract database workflow. For the rendering workflow we were able to run the code up to 131K cores and render images of slices and isosurfaces from the simulation every 5 time steps to produce a visualization of a turbulent mixing layer of 2 fluids. In another of our experiments, we extracted surface-based results, saving the geometry plus field data to our XDB format for later post-processing within FieldView. We saved XDB extracts from every 5th time step, taking around 2% of the simulation runtime. Each XDB file was over 200x smaller than the corresponding volume data file. We wrote 20 XDB files for every volume output file and the combined size of 20 XDB files was still 10x smaller than a single volume output file.

This work is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research under Award Number DE-SC0012449.


Libsim Improvements to Enable Better In Situ Workflows

This year’s Department of Energy Computer Graphics Forum meeting in Pacific Grove, CA, brought together leading visualization experts from the DOE and DOD to share their experiences developing state of the art software needed to analyze results from future exascale computers. The meeting consisted mainly of invited talks spanning a broad set of topics, including: advances in display wall technology, vendor libraries that maximize performance using hardware, software updates, realistic rendering and in situ analysis.

In my talk, “Libsim Improvements to Enable Better In Situ Workflows”, I outlined the significant reductions in both data size and time spent processing the data that can be realized by extracting surfaces of interest and saving data to the XDB format. These XDB files can then be read into FieldView or XDBview.

Additional performance benefits of this workflow are gained due to the fact that subsequent post-processing does not involve reading large amounts of volume-based results. The performance benefits are magnified when the workflow is applied in situ because the data extraction can be done while data are in the solver memory as opposed to being done after writing volume data to disk. In situ workflow sidesteps the I/O bottleneck associated with writing (and later reading) large amounts of data since it restricts data to only the features of interest, which are a small subset of the original data.

My talk demonstrated Intelligent Light’s commitment to in situ and highlighted the improvements that we have made to Libsim, the VisIt in situ library, that enable it to scale to over 131K cores using the AVF-LESLIE combustion code on the Titan supercomputer at Oak Ridge National Laboratory. We have made numerous enhancements to Libsim that improve its efficiency and ability to seamlessly accept data from the host solver code. For instance, we made enhancements that permit zero-copy passing of data from the solver to Libsim when data are not organized contiguously in memory. In addition, we eliminated several bottlenecks that affected VisIt’s rendering and scaling performance on the Titan machine. We also streamlined the creation of XDB files by developing a prototype parallel XDB library based on Oak Ridge’s high performance ADIOS framework. Taken together, these improvements to Libsim and VisIt set the stage for even larger in situ runs to come and eliminate many barriers to using in situ and an extract-based workflow.


Leading the Way With In Situ Extracts

At Intelligent Light, we continue to lead the charge for the adoption of in situ, a technique that can maximize insight from simulation runs while also avoiding the problems caused by saving, storing and moving massive amounts of data. Our work this year shows that using in situ allows the CFD practitioner to increase resolution by saving data at a higher frequency, while still saving far less data overall. This reduces disk space and time to read the data in the post-processing phase.

At the AIAA SciTech 2016 conference this month, I shared our in situ work with the AIAA community in two ways: I presented a paper to the MVCE technical committee titled, "In Situ Infrastructure Enhancements for Data Extract Generation", and I presented an in-booth talk about how to add in situ processing into a solver.

Many of the engineers I met at this year's SciTech are running codes at scale on high performance computers but find it impractical, often impossible, to save all of the data on such systems. In situ enables operations such as visualization and analysis, which have traditionally been performed as post-processing, to be executed in the solver itself as it runs. Instead of writing large amounts of volume data, in situ enables the creation of smaller data products such as images and FieldView XDB extract files. XDB files, for example, capture surfaces of interest as well as scalar and vector fields from the solver and write that data in a compact form orders of magnitude smaller than the standard results file.

GT Rotor visualization. Iso surfaces of Q, colored by Cp. Bottom left includes a cross plane of the mesh.

The paper I presented to the MVCE technical committee, "In Situ Infrastructure Enhancements for Data Extract Generation", describes enhancements made by Intelligent Light to VisIt/Libsim that improve its support for batch-creation of VisIt plots, which can then be exported as XDB extracts. Working with James Forsythe of the US Navy's NAVAIR, the CREATE-AVTM Kestrel solver was instrumented with the latest VisIt/Libsim enhancements for batch support and parallel data writing. Kestrel was run at scales up to 1024 cores using a workflow that produced XDB files every 5 solver iterations, an output frequency far higher than would be attempted with volume-based outputs. Even with writing extracts so often, the in situ production of extract files consumed less than 3% of the overall solver runtime. The set of extract files for a single time step is also 21 times smaller than the corresponding volume data, saving both disk space and time to read in large files for subsequent visualization. Several instances of FieldView operating concurrently processed the resulting XDB files into a movie showing helicopter rotor vortices. One strength of this workflow is that it is parallel from data extraction to extract I/O, all the way through XDB visualization. In addition, the workflow is flexible because XDB extracts provide both geometry and fields that can be visualized, enabling fast data analysis that skips the burden of large I/O using volume data.

Intelligent Light's recent VisIt/Libsim improvements make the process of instrumenting a simulation for in situ simpler than ever before. During the SciTech exhibition, I held a talk in the Intelligent Light booth about how to add in situ processing into a solver. The presentation was well attended by users and solver developers from the US, Japan and Israel. There was much interest in adding VisIt/Libsim and XDB data extraction to solvers and the workflow continues to prove its value.


Vote of Confidence

 The Department of Energy (DOE) again recognized the value of Intelligent Light’s efforts to support innovation by awarding us a Phase IIB SBIR follow-on grant to continue promising R&D on integrating FieldView and VisIt. This brings the total to $2 million that has been committed to enable FieldView to use VisIt’s scalable back end server. Bringing FieldView and VisIt together will empower FieldView users across many disciplines to gain useful insights from the largest datasets generated on the largest computers. The FieldView-VisIt integration extends FieldView’s power into the High Performance Computing (HPC) regime and brings to bear exciting technologies from VisIt such as scalable rendering. Intelligent Light’s success during the Phase II SBIR grant has translated into useful improvements to the VisIt code today and there is more to come during the Phase IIB.

Whereas prior work on FieldView-VisIt integration focused on initial coupling techniques that allow the codes to exchange data, the new work seeks to address performance of the coupling as well as the performance of VisIt itself.  In the early days of VisIt development at LLNL, we had a lot of pressure to add features as opposed to making those features work with the utmost efficiency. This means that there are a lot of places where VisIt can be sped up considerably and otherwise improved.

Related: DOE Invites Intelligent Light to Present In Situ with VisIt, Libsim, & FieldView

Performance improvements are one of the main objectives in the new work. Some of that performance will come from better utilization of parallel resources. For instance, processing an ensemble of datasets or multiple time steps can be achieved through changes in how VisIt handles the data. We plan to make changes to VisIt’s core infrastructure that enable it to process multiple datasets simultaneously in parallel so we can use more compute cores to handle a lot of intermediate sized data. These large modifications will be challenging but we know that the DOE selected Intelligent Light for our ability to carry out demanding work like this, which will benefit the larger VisIt community.

On a personal note, this will be my first time as Principal Investigator on a project of this scale. I have been a VisIt developer from the start and a figure in the VisIt community so this is a great chance for me to continue making important contributions to a code I am passionate about.