Eilmer FAQs

1. Meta-level: why this toolkit?

1.1. Should I use Eilmer4 or Eilmer3?

In our biased opinion, Eilmer v4.0 is all-round better than Eilmer3 so there should be some benefit to updating your input scripts if you were an Eilmer3 user.

The list of supported features in Eilmer v4.0 is hosted at: https://gdtk.uqcloud.net/docs/eilmer/releases/#version-400

1.2. I really want to use Eilmer3. Where do I find it?

Eilmer3 is still available in the Mercurial repository https://source.eait.uq.edu.au/hg/cfcfd3

When working on this website, we had an accident with the old CFCFD website and lost the copy that was being served. The source is all in the mercurial repository so you can still get to all of the instructions. For example, the "getting started" page is at https://source.eait.uq.edu.au/hg/cfcfd3/file/tip/doc/sphinx/getting-started.rst

2. Build and install process

2.1. What do these terms mean: update, clean, build, compile, install?

As developers we take these terms for granted since we might rebuild the code multiple times a day. Here’s what you need to know.

update

If we suggest you update the code, we are referring to grabbing the latest copy of the source code. If you’re working in rolling release mode, you can get this update via a git pull.

clean

Cleaning refers to removing all of the build objects that accumulate in the source directory during a build. We use the name clean as a target in our makefiles. You can execute a clean step with the command make clean.

compile

This typically refers to using the compiler to build small pieces of machine-ready instructions. These pieces could be object files that do nothing on their own, or executables, which are individual working programs.

build

A "build" usually refers to compiling all of the necessary pieces to make up the toolkit of executables.

install

Installing is the process of taking the interesting build pieces (executables, libraries, auxiliary data files) and placing them in a common installation directory.

2.2. How is the build process managed?

We use the Make build system (specifically, GNU Make). This is a system for automating software builds that has its origins in early Unix systems, dating back to 1976. The Make system takes its directions for building from makefile s. The default name is makefile and you will see these files in various source directories across the source tree.

The roles of the makefiles are to make selections about the build process and provide a recipe for the build. In our builds, those selections include:

  • which D compiler to use;

  • which parts of the executable suite to build;

  • which "flavour" (debug or fast) to build (which affects optimisation of the executable)

  • which internal features to enable/disable such as multi-species options and complex numbers mode.

2.3. Why is there a choice between D compilers for the build? Which should I use?

The D language community have a few compiler options available to them. These include the reference compiler dmd (by Digital Mars), and ldc built on top of the LLVM core libraries.

For Eilmer builds, users can choose between dmd and ldc. The reason for choice is partly historical, partly for flexibility, partly for development concerns and partly for optimisation.

When we first began development only dmd was available. Later, the ldc compiler became a reliable tool also. (There is a GNU D compiler, gdc, but I have had limited success with that compiler in recent years.)

Since D is relatively new language, we occasionally bump up against edge cases for language features. Having access to multiple compilers is a good way of checking that we are sticking to mostly well-trodden language features. When we write code that fails to build, it is good to test on a different compiler to see if we’re abusing a language feature or we’ve just run up against bugs in the compilers themselves. This is what I mean by development concerns.

The dmd compiler is the reference compiler but doesn’t give the fastest executing code. (It’s very good, but ldc compiled code is faster.) This is a choice of the dmd team. Their focus is on providing a reference compiler implementation, and they choose not to emphasise internal code optimisations. That’s not to say the dmd compiler can’t do optimised builds, just that it’s not an emphasis. On the other hand, the LLVM project does have a keen focus on the performance of the executable code. For this reason, ldc is recommended for the optimised executable builds. This is why it’s our recommendation for use on HPC where performance matters on a shared resource.

2.4. Where’s a list of the build options for the makefile?

The best response is: "use the source". However, here is a list of build options with explanations as at 25 Feb 2021. This is not an exhaustive list because some options are related to very special cases.

High-level options are:

DMD

selection of D compiler. Default is ldc2, other option is dmd.

FLAVOUR

Options are debug (default) for a debug build, or fast for an optimised build.

PLATFORM

Options are linux (default) or macosx.

INSTALL_DIR

Location for installation files (default: $HOME/dgdinst)

Options related to feature enabling:

WITH_MPI

0 (default) for no MPI version; 1 to include the MPI build.

WITH_MPI_TIMEOUTS

0 (default) for no internal timeout checking (use e4monitor); 1 to turn on internal checking for timeouts

WITH_NK

0 (default) for exclusion of Newton-Krylov accelerator; 1 to include it.

WITH_LUSGS

0 (default) for exclusion of lower-upper symmetric-Gauss-Seidel update; 1 to include it

WITH_SSC

0 (defualt) for exclusion of shape-sensitivity core functions for adjoint work; 1 to include them

WITH_OPENCL_GPU_CHEM

0 (default) for exclusion of GPU chemistry module implemented in OpenCL; 1 to include it

WITH_CUDA_GPU_CHEM

0 (default) for exclusion of GPU chemistry module implemented with CUDA; 1 to include it

WITH_COMPLEX_NUMBERS

0 (default) for exclusion of complex number mode; 1 to turn it on

WITH_FPE

0 (default) for no trapping of floating-point exceptions; 1 to turn it on and halt on floating-point exceptions

WITH_DVODE

0 (default) for exclusion of DVODE Fortran ODE library; 1 to include it

WITH_MATPLOTLIB

0 (default) for exclusion of Matplotlib library calls; 1 to include it

MULTI_SPECIES_GAS

1 (default) to allow for multiple-species simulations; 0 to restrict to single species only

MULTI_T_GAS

1 (default) to allow for multiple temperatures; 0 to restrict to single temperature only

MHD

1 (default) to include modelling terms for magnetohydrodynamics; 0 to disable those modelling terms

TURBULENCE

1 (default) to include RANS turbulence model terms; 0 to disable that modelling

WITH_THREAD_SANITIZER

0 (default) — CHECK WITH DEV TEAM.

2.5. This all seems a bit confusing. What are the recommendations?

The recommendation depends on your how you want to use the code. Here are some scenarios.

2.5.1. I’d like a simple build to try things out on my laptop or desktop.

We recommend a default build and install. Try this:

$ cd dgd/src/eilmer
$ make install
for use on a cluster computer (with MPI)

Sounds like you want an optimised build and MPI. In the Eilmer source directory, do this:

$ make DMD=ldc2 FLAVOUR=fast WITH_MPI=1 install

2.5.3. I’m an expert. I know exactly what modelling options I want, and I’d like to reduce the memory footprint.

Say you had a laminar flow, a single species and single temperature, you could really optimise the selections by doing:

$ make DMD=ldc2 FLAVOUR=fast WITH_MPI=1 TURBULENCE=0 MULTI_SPECIES_GAS=0 MULTI_T_GAS=0 MHD=0 install

2.6. This is overwhemling. Isn’t there a script that would just take care of this build and install process for me?

Yes, there are several scripts available to help you. Take a look in dgd/install-scripts. The script names are self-describing.

2.7. I’m using a RHEL 8 system and the packaged OpenMPI doesn’t seem to work with the Eilmer build. What can I do?

You might need to experiment with a different version of OpenMPI compared to that maintained in the RHEL repositories. We’ve had reports that opempi-4.0.3 works. You can install it using a src rpm. The following instructions were provided by Jeremy Moran.

  1. Download openmpi-4.0.3 as a src rpm.

  2. Build

    $ rpmbuild --rebuild openmpi-4.0.3-1.src.rpm
  3. Install

    $ sudo yum localinstall openmpi-4.0.3-1_x86_64.rpm

2.8. The recommendation is for the latest D compiler, but that doesn’t play nicely with glibc on the system. What can I do?

While we typically recommend the latest D compiler, you can use other fairly recent versions to build our code.

For example, a user reported success with ldc2 v1.26.0 on CentOS 7.4 system. (For UQ people, that system is tinaroo.)

3. Problems on cluster computers

4. Units of input/output

4.1. In axisymmetric simulations, the outputs include areas and volumes. How are these defined?

When using axisymmetric mode, a useful mental model is to think of the simulation domain as a sector of a circle because this is the assumption we have applied in the implementation. The included angle in the sector is one radian.

So that answers the question. The areas and volumes are defined per radian. If you need the effective area over the full (axisymmetric) geometry, multiply by 2$\pi$.

5. Thermochemistry questions

5.1. When preparing a thermally perfect gas model for use, I get a warning about missing CEA coefficients for viscosity and thermal conductivity for some species. Should I be worried? Why does this happen?

When preparing a thermally perfect gas model, Eilmer defaults to attempting to use the CEA coefficients for thermodynamics and transport properties. The thermodynamic coefficients have been taken from the CEA file thermo.inp, and the transport property coefficients are from trans.inp. If you look in trans.inp, you’ll notice that is has data for far fewer species than those listed in thermo.inp. In other words, the transport property data was not available or not important for the builders of CEA. So this answers the question why does this happen. It happens because the data is simply not available from CEA. What the prep-gas program will do is supply default values from the defaults.lua file. You can inspect that to see what the defaults are. They are most likely the properties for air.

The other question, "should I be worried?", has a more complex answer. It depends. If you are doing an inviscid simulation, then there’s nothing to be concerned about. This warning is related to transport properties. These only come into play for viscous simulations. If you are doing a viscous simulation, then you need to apply some judgement. Are these species with missing transport data minor species in the mixture? If so, their contribution to the bulk viscosity and thermal conductivity is probably minor. If this is the case, it would probably be quite acceptable to use the substituted air properties for these minor species.

But no, these missing species are really important to me?

Sounds like you’re doing a combustion problem. Usually these missing species properties arise for intermediate species in combustion processes. In that case, you’ll probably want to dip into the Grimech database instead. It is often more complete for these species. That is also available in Eilmer for many (but not all) species. You can configure this option in your input file for prep-gas. Add the following line to your input file:

options = {database='prefer-grimech'}

My species are still missing when I use the Grimech database!!!

Well, now it sounds like you’ll need to hunt down the data for the transport properties yourself. You can add them to your prepared gas model input file, or better yet, add them to the source code and send us your contribution. There is information on adding complete species and species data in: dgd/src/gas/species-database/README.rst.

6. Generic WTF?

6.1. What is WC, WCtFT and WCtMS?

Wall Clock, Wall Clock till Final Time and Wall Clock till Maximum allowed Steps estimated in seconds.