// @HEADER // ***************************************************************************** // Phalanx: A Partial Differential Equation Field Evaluation // Kernel for Flexible Management of Complex Dependency Chains // // Copyright 2008 NTESS and the Phalanx contributors. // SPDX-License-Identifier: BSD-3-Clause // ***************************************************************************** // @HEADER #ifndef PHX_DOXYGEN_DOCUMENTATION_HPP #define PHX_DOXYGEN_DOCUMENTATION_HPP /*! \mainpage \section main_index Index - \ref overview - \ref news - \ref user_guide - \ref faq - \ref bugs - \ref history - \ref authors - \ref copyright - \ref questions \section overview Overview Phalanx is a local graph-based field evaluation toolkit. While the intended use case is for solving general partial differential equations (PDEs), there is NO specific code implemented for PDEs in Phalanx. It can be applied to any system that requires function evaluation. In terms of PDE discretization schemes it can be used for finite element, finite difference and finite volume. Phalanx is a local node evaluation tool. Phalanx relies on the Kokkos package for performance portability and provides a simple performant interface to multicore and manycore architectures. While its main use is for large scale parallel high performance computing, the MPI communication (possibly required for ghosting) must be handled by other packages in the toolchain (separation of concerns). Users can handle this manually as Phalanx places no requirements on this but we recomend the Tpetra package in Trilinos be leveraged. The main goal of Phalanx is to decompose a complex problem into a number of simpler problems with managed dependencies to support rapid development and extensibility. Through the use of template metaprogramming concepts, Phalanx supports arbitrary user defined data types and evaluation types. This allows for extreme flexibility in integration with user applications and provides extensive support for embedded technology such as automatic differentiation for sensitivity analysis, optimization, and uncertainty quantification. This approach, coupled with the template capabilities of C++ offers a number of unique and powerful capabilities: Phalanx is a hammer. It's use should be carefully considered. We recommend its use when writing a general PDE framework where one needs support for flexibility in equation sets, discretizations, and material models. It should not be used for a fixed set of equations and a single discretization that never changes (Although the template metaprogramming support could still be leveraged). There are some drawbacks to using Phalanx that should be considered: - A potential performance loss due to fragmentation of the over-all algorithm (e.g., several small loops instead of one large loop). Since the granularity of the directed acyclic graph is controlled by the user, a poor choice in functional decomposition is possible. This could result in inefficient use of memory and costly kernel launches if there is little work in the evaluator. - A potential loss of visibility of the original, composite problem (since the code is scattered into multiple places). Managing these trade-offs can result in application code that both performs well and supports rapid development and extensibility. \section news News Phalanx has just completed the transition from shards::Array to the Kokkos::View. Leveraging Kokkos, Phalanx is now performance portable to GPUs. The examples and user guide, however, have not been updated to reflect the transition. Please look at the MultiDimensionalArray example and the unit tests for examples on how to use Phalanx. We will update the user guide and other examples asap. \section bugs Reporting Bugs and Making Enhancement Requests To reports bugs or make enhancement requests, visit the Trilinos Bugzilla (Bug Tracking) Database, and use the following instructions. \section history History Phalanx grew out of the Variable Manger in the Charon code and the Expression Manager in the Aria code at Sandia National Laboratires. It is an attempt at merging the two capabilities. \section authors Authors and Contributors The following have contributed to the design through ideas, discussions, and/or code development of Phalanx: - Roger Pawlowski (Lead Developer), SNL - Irina Demeshko, SNL - Eric Phipps, SNL - Pat Notz, SNL \section copyright Copyright \verbinclude copyright.txt \section questions For All Questions and Comments... Please contact Roger Pawlowski (rppawlo@sandia.gov). */ /* ************************************************************************ */ /* ************************************************************************ */ /*! \page user_guide Users Guide \section user_guide_index Index - \ref user_guide_getting_started - \ref user_guide_domain_model - \ref user_guide_mdarray_domain_model - \ref performance - \ref user_guide_step1 - \ref user_guide_step2 - \ref user_guide_step3 - \ref user_guide_step4 - \ref user_guide_step5 - \ref user_guide_step6 - \ref questions \section user_guide_getting_started Getting Started \subsection ug_dummy_1 A. Understand Templates Phalanx is a complex package that make heavy use of the C++ templating mechanism. We recommend that users of the Phalanx package first familiarize themselves with C++ templates. An excellent reference is "C++ Templates: The Complete Guide" by Vandevoorde and Josuttis, 2003. While users do not need to understand template metaprogramming to use Phalanx, the concepts underlying many operations in Phalanx can be found in "C++ Template Metaprogramming" by Abrahams and Gurtovoy, 2004 and the Boost template metaprogramming library (MPL). Once Phalanx is integrated into a code by a template savvy developer, the only operation application users should perform is to extend the evaluation routines to new equations and new models by adding new Evaluators. We realize that application developers and application users typically have distictly different skill sets. Much design work has been invested in making the Evaluator implementation very clean and as template free as possible. Therefore, users of the code who only write new Evaluators DO NOT need to know templates. \subsection ug_dummy_2 B. Learn the Phalanx Nomenclature Users should then learn the nomeclature used in the package defined in the \ref user_guide_domain_model. \subsection ug_dummy_3 C. Tutorial The main concept of Phalanx is to evaluate fields typically for solving PDEs (although it is not limited to this). We demonstrate the integration process using the simple example found in phalanx/example/MultiDimensionalArray. Suppose that we want to solve the heat equation over the physical space \f$ \Omega \f$: \f[ \nabla \cdot (-k \nabla T) + s = 0 \f] where \f$ T \f$ is the temparature (and our degree of freedom), \f$ k \f$ is the thermal conductivity, and \f$s\f$ is a nonlinear source term. We pose this in terms of a conservation law system: \f[ \nabla \cdot (\mathbf{q}) + s = 0 \f] where \f$ \mathbf{q} = -k \nabla T \f$ is the heat flux. The specific discretization technique whether finite element (FE) or finite volume (FV) will ask for \f$\mathbf{q}\f$ and \f$s\f$ at points on the cell. Phalanx will evaluate \f$\mathbf{q}\f$ and \f$s\f$ at those points and return them to the discretization driver. Using finite elements, we pose the problem in variational form: Find \f$ u \in {\mathit{V^h}} \f$ and \f$ \phi \in {\mathit{S^h}} \f$ such that: \f[ - \int_{\Omega} \nabla \phi \cdot \mathbf{q} d\Omega + \int_{\Gamma} \phi \mathbf{n} \cdot \mathbf{q} + \int_{\Omega} \phi s d\Omega = 0 \f] Phalanx will evaluate \f$\mathbf{q}\f$ and \f$s\f$ at the quadrature points of the cells and pass them off to the integrator such as Intrepid. This is a trivial example, but the dependency chains can grow quite complex if performing something such as a chemically reacting flow calculation coupled to Navier-Stokes and energy conservation. Follow the steps below to integrate Phalanx into your application. The example code shown in the steps comes from the energy flux example in the directory "phalanx/example/EnergyFlux". Note that many classes are named with the word "My" such as MyWorkset, MyTraits, and MyFactory traits. Any object that starts with the word "My" denotes that this is a user defined class. The user must implement this class specific to their application. All Evaluator derived objects are additionally implemented by the user even though they do not follow the convention of starting with the word "My". - \ref user_guide_step1 - \ref user_guide_step2 - \ref user_guide_step3 - \ref user_guide_step4 - \ref user_guide_step5 - \ref user_guide_step6 \section user_guide_domain_model Phalanx Domain Model \section performance Performance Some recomendations for efficient code: \section user_guide_mdarray_domain_model Multi-Dimensional Array Domain Model The multidimensional array was designed for interoperability between a number of Trilinos packages including shards, phalanx and intrepid. These codes each have their own implementations but follow a basic set of requirements so that functions templated on the array type can use any of the multidimensional array implementations. The required functions are: Phalanx implements the multidimensional array in the PHX::MDField class and supports arrays with up to 8 ranks (one more than Fortran arrays support). More information can be found in the shards library. \section user_guide_step1 Step 1: Configuring, Building, and installing Phalanx \subsection ug_step1_general A. General Library Requirements Phalanx is distributed as a package in the Trilinos Framework. It can be enabled as part of a trilinos build with the configure option "-D Trilinos_ENABLE_Phalanx=ON". Phalanx currently has direct dependencies on the following third party libraries: - Requires the Teuchos utilities library, part of the Trilinos Framework. This will automatically be enabled when you enable the phalanx library. - Requires the Sacado Automatic Differentiation Library, part of the Trilinos Framework. This will automatically be enabled when you enable the phalanx library. \subsection ug_step1_performance B. Performance Example Requirements - Optional: Some performance tests run comparisons against TVMET: Tiny Vector Matrix library using Expression Templates. This is to get a feel for how our "dumb" vector matrix objects perform compared to expression templates. You must enable the tvmet TPL add the path to the TVMET library during Trilinos configuration. An example configuration file can be found in Trilinos/packages/phalanx/build_scripts/build_phalanx_gcc.sh. TVMET is optional and hidden behind an ifdef. The performance tests will be built regardless of whether tvmet is enabled/disabled. \subsection ug_step1_fem C. Nonlinear Finite Element Example Requirements To build the example problem in "phalanx/example/FEM_Nonlinear", distributed vector, distributed sparse matrix, and corresponding linear solvers are required. The following Trilinos packages need to be enabled to build the FEM_Nonlinear example: - Requires: the Epetra Library, supplies the linear algebra data structures for spares matrices. Can be enabled during the Trilinos configure with the flag "-D Trilinos_ENABLE_Epetra=ON". - Requires: the Ifpack Library, supplies incomplete factorization preconditioners. Can be enabled during the Trilinos configure with the flag "-D Trilinos_ENABLE_Ifpack=ON". - Requires: the Belos Library, supplies block GMRES iterative linear solver. Can be enabled during the Trilinos configure with the flag "-D Trilinos_ENABLE_Belos=ON". This example will be disabled if the above packages are not enabled. \subsection db D. Configure Trilinos/Phalanx The general instructions for building trilinos can be found at Trilinos Documentation Page. Of particular importance are the Overview, User Guide, and Tutorial documents. At a minimum you must enable the Teuchos, Sacado, and Phalanx packages. An example configure script is: \verbinclude reconfigure.linux Once configure is run, build and install the library with the command: \code make install \endcode \section user_guide_step2 Step 2: Determine Types Users must next determine the evaluation types and data types they will require in their simulation. Please see the section \ref user_guide_domain_model for detailed explanation of types. Following the example in the phalanx/example/EnergyFlux directory, we will be requiring two evaluation types, one for the residual evaluation of the discretized PDE equation and one for the corresponding Jacobian. Additional evaluation types might be for the parameter sensitivites and uncertainty quantification. Once the evaluation types are chosen, users must decide on a default scalar type and on all data types that are valid for each evaluation type. The data types should all be templated on the default scalar type for the particular evaluation type, but this is not a requirement (expert users can violate this for performance reasons). Uses must implement their own data types or get them from a separate library. For sensitivities, the trilinos package Sacado should be used. For uncertainty quantification, the Trilinos package Stokhos should be used. In our example, the user has written an implementation of vector (MyVector) and matrix (MyTensor) classes found in the file AlgebraicTypes.hpp. They are templated on a scalar type and can be used for all evaluation types. Typical examples of algebraic types include vectors, matrices, or higher order tensor objects, but in reality can be any struct/class that the user desires. Remember to template the objects on the scalar type if possible. An example of a very inefficient Vector and Matrix implementation (operator overloading without expression templates) can be found in AlgebraicTypes.hpp. \section user_guide_step3 Step 3: Write the Traits Object The Traits object is a struct that defines the evaluation types, the data types for each evaluation type, the allocator type, and the user defined types that will be passed through evaluator calls. We will go through each of these defninitions. The basic class should derive from the PHX::TraitsBase object. The Traits struct must define each of the following typedef members of the struct: - \b EvalTypes - an mpl::vector of user defined evaluation types. Each evaluation type must have a typedef member called ScalarT that provides the default scalar type. This is used to automate the building of evaluators for each evaluation type using the EvaluatorFactory. - \b EvalToDataMap - an mpl::map. The key is an evaluation type and the value is an mpl::vector of valid data types for that particular evaluation type. - \b Allocator type - type that defines the allocator class to use to allocate the memory for data storage. - \b SetupData - A user defined type to be passed in to the postRegistrationSetup() call. Allows users to pass in arbitrary data to the evaluators during setup. - \b EvalData - A user defined type to be passed in to the evaluateFields() call. Allows users to pass in arbitrary data. - \b PreEvalData - A user defined type to be passed in to the preEvaluate() call. Allows users to pass in arbitrary data. - \b PostEvalData - A user defined type to be passed in to the postEvaluate() call. Allows users to pass in arbitrary data. \subsection td1 A. Basic Class The basic outline of the traits struct is: \code struct MyTraits : public PHX::TraitsBase { . . . }; \endcode Inside this struct we need to implement all the typedefs listed above. The example we will follow is in the file Traits.hpp in "phalanx/example/EnergyFlux" directory. \subsection td2 B. EvalTypes First we need to know the evaluation types. Each evaluation type must include at least a typedef'd default scalar type argument as a public member called ScalarT. Here is the example code: \code struct MyTraits : public PHX::TraitsBase { // ****************************************************************** // *** Scalar Types // ****************************************************************** // Scalar types we plan to use typedef double RealType; typedef Sacado::Fad::DFad FadType; // ****************************************************************** // *** Evaluation Types // ****************************************************************** struct Residual { typedef RealType ScalarT; }; struct Jacobian { typedef FadType ScalarT; }; typedef Sacado::mpl::vector EvalTypes; . . . \endcode The typedefs RealType and FadType are done only for convenience. They are not actually required but cut down on the typing. Only the EvalTypes typedef is required in the code above. \subsection td3 C. EvaltoDataMap Next we need to link the data types to the evaluation type. Note that one could use the same data type in multiple evaluation types: \code . . . // Residual (default scalar type is RealType) typedef Sacado::mpl::vector< RealType, MyVector, MyTensor > ResidualDataTypes; // Jacobian (default scalar type is Fad) typedef Sacado::mpl::vector< FadType, MyVector, MyTensor > JacobianDataTypes; . . . \endcode \subsection td4 D. Allocator Define the Allocator type to use. Phalanx comes with two allocators, but the user can write their own allocator class if these aren't sufficient. The Phalanx allocator classes are: - PHX::NewAllocator: uses the C++ "new" command to allocate each field on the heap separately. - PHX::ContiguousAllocator: allocates a single contiguous block of memory on the heap for all fields regardless of the type. This allows us to fit a subset of elements into cache to speed up the evaluation. Code using the NewAllocator is: \code . . . // ****************************************************************** // *** Allocator Type // ****************************************************************** typedef PHX::NewAllocator Allocator; . . . \endcode \subsection td5 E. EvalData,PreEvalData,PostEvalData Users can pass their own data to the postRegistrationSetup(), evaluateFields(), preEvaluate() and postEvaluate() methods of the PHX::FiledManager class. In this example, the user passes in a struct that they have written called MyEvalData. This contains information about the cell workset. The user is not required to write their own object. They could just pass in a null pointer if they don't need auxiliary information passed into the routine. This is demonstrated in the SetupSetup, PreEvalData, and PostEvalData. A void* is set for the data member. \code . . . // ****************************************************************** // *** User Defined Object Passed in for Evaluation Method // ****************************************************************** typedef void* SetupData; typedef const MyEvalData& EvalData; typedef void* PreEvalData; typedef void* PostEvalData; }; \endcode \section user_guide_step4 Step 4: Specialize the PHX::TypeString Object For debugging information, Phalanx makes a forward declaration of the PHX::TypeString object. This must be specialized for each evaluation type and each data type so that if there is a run-time error, phalanx can report detailed information on the problem. We could have used the typeinfo from the stl, but the name() method is not demangled on every platform, so it can make debugging a challenge. The specialized classes can go into their own file or can be added to the traits file above depending on how you use the Traits class. During linking, if the compiler complains about multiple defninitions of your specialized traits classes, separate the traits implementation into their own .cpp file. \code namespace PHX { // ****************************************************************** // ****************************************************************** // Debug strings. Specialize the Evaluation and Data types for the // TypeString object in the phalanx/src/Phalanx_TypeString.hpp file. // ****************************************************************** // ****************************************************************** // Evaluation Types template<> struct TypeString { static const std::string value; }; template<> struct TypeString { static const std::string value; }; const std::string TypeString::value = "Residual"; const std::string TypeString::value = "Jacobian"; // Data Types template<> struct TypeString { static const std::string value; }; template<> struct TypeString< MyVector > { static const std::string value; }; template<> struct TypeString< MyTensor > { static const std::string value; }; template<> struct TypeString< Sacado::Fad::DFad > { static const std::string value; }; template<> struct TypeString< MyVector > > { static const std::string value; }; template<> struct TypeString< MyTensor > > { static const std::string value; }; const std::string TypeString::value = "double"; const std::string TypeString< MyVector >::value = "MyVector"; const std::string TypeString< MyTensor >::value = "MyTensor"; const std::string TypeString< Sacado::Fad::DFad >:: value = "Sacado::Fad::DFad"; const std::string TypeString< MyVector > >:: value = "Sacado::Fad::DFad< MyVector >"; const std::string TypeString< MyTensor > >:: value = "Sacado::Fad::DFad< MyTensor >"; } \endcode \section user_guide_step5 Step 5: Write your Evaluators Before Writing your evaluators, you must decide on how you plan to build the evaluators. In most cases you will want to build one Evaluator for each evaluation type. Phalanx provides an automated factory called the PHX::EvaluatorFactory that will build an evaluator for each evaluation type automatically, but this places a restriction on the constructor of all evaluators built this way. If you plan to use the automated builder, the constructor for the Evaluator must contain only one argument - a Teuchos::ParameterList. This paramter list must contain a key called "Type" with an integer value corresponding to the type of Evaluator object to build. The parameterlist can contain any other information the user requires for proper construction of the Evaluator. You are not restricted to using the automated factory for every Evaluator. You can selectively use the automated factory where convenient. For each field, you will need an evaluator. Evaluators can evlauate multiple fields at the same time. You can derive from the base class PHX::Evaluator, or you can derive from class PHX::EvaluatorWithBaseImpl that has most of the methods already implemented so that the same support code is not replicted in each evaluator. We STRONGLY recommend deriving from the class PHX::EvaluatorWithBaseImpl. An example for evaluating the density field is: \code #ifndef PHX_EXAMPLE_VP_DENSITY_HPP #define PHX_EXAMPLE_VP_DENSITY_HPP #include "Phalanx_config.hpp" #include "Phalanx_Evaluator_WithBaseImpl.hpp" #include "Phalanx_Evaluator_Derived.hpp" #include "Phalanx_Field.hpp" template class Density : public PHX::EvaluatorWithBaseImpl, public PHX::EvaluatorDerived { public: Density(const Teuchos::ParameterList& p); void postRegistrationSetup(typename Traits::SetupData d, PHX::FieldManager& vm); void evaluateFields(typename Traits::EvalData ud); private: typedef typename EvalT::ScalarT ScalarT; double constant; PHX::Field density; PHX::Field temp; std::size_t data_layout_size; }; #include "Evaluator_Density_Def.hpp" #endif \endcode Note that if you want to use the automated factory PHX::EvaluatorFactory to build an object of each evaluation type, you must derive from the PHX::EvaluatorDerived class as shown in the example above. This allows the variable manager to store a vector of base object pointers for each evaluation type in a single stl vector. Also note that we pull the scalar type, ScalarT, out of the evaluation type. The implementation is just as simple: \code // ********************************************************************** template Density:: Density(const Teuchos::ParameterList& p) : density("Density", p.get< Teuchos::RCP >("Data Layout") ), temp("Temperature", p.get< Teuchos::RCP >("Data Layout") ) { this->addEvaluatedField(density); this->addDependentField(temp); this->setName("Density"); } // ********************************************************************** template void Density:: postRegistrationSetup(typename Traits::SetupData d, PHX::FieldManager& vm) { this->utils.setFieldData(density,vm); this->utils.setFieldData(temp,vm); data_layout_size = density.fieldTag().dataLayout().size(); } // ********************************************************************** template void Density::evaluateFields(typename Traits::EvalData d) { std::size_t size = d.num_cells * data_layout_size; for (std::size_t i = 0; i < size; ++i) density[i] = temp[i] * temp[i]; } \endcode The constructor pulls out data from the parameter list to set the correct data layout. Additionally, it tells the FieldManager what fields it will evaluate and what fields it requires/depends on to perform the evaluation. The postRegistrationSetup method gets pointers from the FieldManager to the array for storing data for each particular field. Writing evaluators can be tedious. We have invested much time in minimizing the amount of code a user writes for a new evaluator. Our experience is that you can literally have hundreds of evaluators. So we have added macros to hide the boilerplate code in each evaluator. Not only does this streamline/condense the code, but it also hides much of the templating. So if your user base is uncomfortable with C++ templates, the macro definitions could be very helpful. The definitions are found in the file Phalanx_Evaluator_Macros.hpp. The same evaluator shown above is now implemented using the macro definitions: Class declaration: \code #ifndef PHX_EXAMPLE_VP_DENSITY_HPP #define PHX_EXAMPLE_VP_DENSITY_HPP #include "Phalanx_Evaluator_Macros.hpp" #include "Phalanx_Field.hpp" PHX_EVALUATOR_CLASS(Density) double constant; PHX::Field density; PHX::Field temp; std::size_t data_layout_size; PHX_EVALUATOR_CLASS_END #include "Evaluator_Density_Def.hpp" #endif \endcode Class definition: \code //********************************************************************** PHX_EVALUATOR_CTOR(Density,p) : density("Density", p.get< Teuchos::RCP >("Data Layout") ), temp("Temperature", p.get< Teuchos::RCP >("Data Layout") ) { this->addEvaluatedField(density); this->addDependentField(temp); this->setName("Density"); } //********************************************************************** PHX_POST_REGISTRATION_SETUP(Density,data,fm) { this->utils.setFieldData(density,fm); this->utils.setFieldData(temp,fm); data_layout_size = density.fieldTag().dataLayout().size(); } //********************************************************************** PHX_EVALUATE_FIELDS(Density,d) { std::size_t size = d.num_cells * data_layout_size; for (std::size_t i = 0; i < size; ++i) density[i] = temp[i] * temp[i]; } \endcode The evaluators for the example problem in "phalanx/example/EnergyFlux" have been rewritten using the macro definitions and can be found in the directory "phalanx/test/Utilities/evaluators". Finally, since writing even the above code contains much boilerplate, we have written a python script that will generate the above skeleton files for you. All you need provide is the class name and the filename. The script is called phalanx_create_evaluator.py and can be found in the "scripts" directory. A "make install" will place the script in the "bin" directory. To generate a skeleton for the above function, you would execute the following command at the prompt: \code > ./phalanx_create_evaluator.py -c -n Density Evaluator_Density \endcode \section user_guide_step6 Step 6: Implement the FieldManager in your code Adding the FieldManager to your code is broken into steps. You must build each Evaluator for each field type, register the evaluators with the FieldManager, and then call the evaluate routines. Continuing from our example: \subsection fmd1 A. Building your Evaluators Users can build their own Evaluators and register them with the FieldManager or they can use our automated factory, PHX::EvaluatorFactory to handle this for them. Normally, users will want to build an evaluator for each evaluation type. The factory makes this very easy. Additionally, you are not restricted to using the automated factory for every Evaluator. You can selectively use the automated factory where convenient. The next two sections describe how to build the evaluators. \subsubsection fmd1s1 A.1 Building and Registering Evaluators Manually. To build an Evaluator manually, all you need to do is allocate the Evaluator on the heap using a reference counted smart pointer (Teuchos::RCP) to point ot the object. This will ensure proper memory management of the object. Here is an example of code to build a Density evaluator for each evaluation type and register it with the corresponding manager. \code // Create a FieldManager FieldManager fm; // Constructor requirements RCP p = rcp(new ParameterList); p->set< RCP >("Data Layout", qp); // Residual Teuchos::RCP< Density > residual_density = Teuchos::rcp(new Density(p)); fm.registerEvaluator(residual_density); // Jacobian Teuchos::RCP< Density > jacobian_density = Teuchos::rcp(new Density(p)); fm.registerEvaluator(jacobian_density); \endcode As one can see, this becomes very tedious if there are many evaluation types. It is much better to use the automated factory to build one for each evalaution type. Where this method is useful is if you are in a class already templated on the evaluation type, and would like to build and register an evaluator in that peice of code. \subsubsection fmd1s2 A.2 Using the Automated Factory Phalanx provides an automated builder PHX::EvaluatorFactory that will create an object of each evaluation type. The following requirements are placed on each and every Evaluator that a user writes if they plan to use the automated factory:
  1. The constructor of your Evaluator must take exactly one argument that is a Teuchos::ParamterList object. A Teuchos::ParameterList allows you to pass an arbitrary number of objects with arbitrary types.
  2. In the Teuchos::ParamterList, you must have one key called "Type" that is associated with an integer value that uniquely corresponds to an Evaluator object written by the user. This integer is defined in the users factory traits object described below.
  3. The Evaluator must derive from the PHX::EvaluatorDerived class as shown in the example above. This allows the variable manager to store a vector of base object pointers for each evaluation type in a single stl vector yet be able to return the derived class.
To build an PHX::EvaluatorFactory, you must provide a factory traits class that gives the factory a list of its evaluator types. An example of the factory traits class is the FactoryTraits object in the file FactoryTraits.hpp: \code #ifndef EXAMPLE_FACTORY_TRAITS_HPP #define EXAMPLE_FACTORY_TRAITS_HPP // mpl (Meta Programming Library) templates #include "Sacado_mpl_vector.hpp" // User Defined Evaluator Types #include "Evaluator_Constant.hpp" #include "Evaluator_Density.hpp" #include "Evaluator_EnergyFlux_Fourier.hpp" #include "Evaluator_FEInterpolation.hpp" #include "Evaluator_NonlinearSource.hpp" #include "Sacado_mpl_placeholders.hpp" using namespace Sacado::mpl::placeholders; template struct MyFactoryTraits { static const int id_constant = 0; static const int id_density = 1; static const int id_fourier = 2; static const int id_feinterpolation = 3; static const int id_nonlinearsource = 4; typedef Sacado::mpl::vector< Constant<_,Traits>, // 0 Density<_,Traits>, // 1 Fourier<_,Traits>, // 2 FEInterpolation<_,Traits>, // 3 NonlinearSource<_,Traits> // 4 > EvaluatorTypes; }; #endif \endcode Since the factory is built at compile time, we need to link a run-time choice to the compile-time list of object types. Thus the user must provide the static const int identifiers that are unique for each type that can be constructed. Users can ignore the "_" argument in the mpl vector. This is a placeholder argument that allows us to iterate over and instert each evaluation type into the factory traits. Now let's build the evaluators. The following code can be found in the Example.cpp file in the directory "phalanx/example/EnergyFlux". We create a ParameterList for each evaluator and call the factory constructor. Since we are using the factory, we need to specify the "Type" argument set to the integer of the corresponding Evaluator in the factory traits that we wish to build: \code RCP qp = rcp(new FlatLayout("QP", 4)); RCP node = rcp(new FlatLayout("NODE", 4)); // Parser will build parameter list that determines the field // evaluators to build map > evaluators_to_build; { // Temperature RCP p = rcp(new ParameterList); int type = MyFactoryTraits::id_constant; p->set("Type", type); p->set("Name", "Temperature"); p->set("Value", 2.0); p->set< RCP >("Data Layout", node); evaluators_to_build["DOF_Temperature"] = p; } { // Density RCP p = rcp(new ParameterList); int type = MyFactoryTraits::id_density; p->set("Type", type); p->set< RCP >("Data Layout", qp); evaluators_to_build["Density"] = p; } { // Constant Thermal Conductivity RCP p = rcp(new ParameterList); int type = MyFactoryTraits::id_constant; p->set("Type", type); p->set("Name", "Thermal Conductivity"); p->set("Value", 2.0); p->set< RCP >("Data Layout", qp); evaluators_to_build["Thermal Conductivity"] = p; } { // Nonlinear Source RCP p = rcp(new ParameterList); int type = MyFactoryTraits::id_nonlinearsource; p->set("Type", type); p->set< RCP >("Data Layout", qp); evaluators_to_build["Nonlinear Source"] = p; } { // Fourier Energy Flux RCP p = rcp(new ParameterList); int type = MyFactoryTraits::id_fourier; p->set("Type", type); p->set< RCP >("Data Layout", qp); evaluators_to_build["Energy Flux"] = p; } { // FE Interpolation RCP p = rcp(new ParameterList); int type = MyFactoryTraits::id_feinterpolation; p->set("Type", type); p->set("Node Variable Name", "Temperature"); p->set("QP Variable Name", "Temperature"); p->set("Gradient QP Variable Name", "Temperature Gradient"); p->set< RCP >("Node Data Layout", node); p->set< RCP >("QP Data Layout", qp); evaluators_to_build["FE Interpolation"] = p; } // Build Field Evaluators for each evaluation type EvaluatorFactory > factory; RCP< vector< RCP > > > evaluators; evaluators = factory.buildEvaluators(evaluators_to_build); // Create a FieldManager FieldManager fm; // Register all Evaluators registerEvaluators(evaluators, fm); \endcode The map "evaluators_to_build" has a key of type "std::string". This key is irrelevant to the execution of the code. It is an identifer that can help users debug code or search for a specific provider in the list. What you put in the key is for your own use. You are free to register evaluators until the time you call the method postRegistrationSetup() on the field manager. Once this is called, you can not register any more evaluators. You can make multiple registration calls to add more providers - there is no limit on the number of calls. \subsection fmd2 B. Request which Fields to Evaluate You must tell the FieldManager which quantities it should evaluate. This step can occur before, after, or in-between the registration of Evaluators. You must request variables separately for each evaluation type. This is required since since data types can exist in multiple evaluation types. \code // Request quantities to assemble RESIDUAL PDE operators { typedef MyTraits::Residual::ScalarT ResScalarT; Tag< MyVector > energy_flux("Energy_Flux", qp); fm.requireField(energy_flux); Tag source("Nonlinear Source", qp); fm.requireField(source); } // Request quantities to assemble JACOBIAN PDE operators { typedef MyTraits::Jacobian::ScalarT JacScalarT; Tag< MyVector > energy_flux("Energy_Flux", qp); fm.requireField(energy_flux); Tag source("Nonlinear Source", qp); fm.requireField(source); } \endcode You are free to request fields to evaluate until the time you call the method postRegistrationSetup() on the field manager. Once this is called, you can not request any more fields. \subsection fmd3 C. Call FieldManager::postRegistrationSetup() Once the evaluators are registered with the FieldManager and it knows which field it needs to provide, call the postRegistrationSetup() method on the FieldManager. This method requires that the user specify the maximum number of cells for each evaluation - the workset size. This number should be selected so that all fields can fit in the cache of the processor if possible. \code // Assume we have 102 cells on processor and can fit 20 cells in cache const std::size_t num_local_cells = 102; const std::size_t workset_size = 20; fm.postRegistrationSetup(workset_size); \endcode The postRegistrationSetup() method causes the following actions to take place in the FieldManager:
  1. Based on the requested fields in \ref fmd2, the FieldManager will trace through the evaluators to determine which evaluators to call and the order in which they need to be called to achieve a consistent evaluation. Not all evaluators that are registered will be used. They will only be called to satisfy dependencies of the required fields.
  2. Once the dependency chain is known, we can pull together a flat list of all fields that will be used. Now the FieldManager will allocate memory to store all fields. It will use the Allocator object from the users traits class to accomplish this. By unifying the allocation into a single object, we can force all fields to be allocated in a single contiguous block of memory if desired.
  3. Once the memory for field data is allocated, we must set the pointer to that memory block inside each Field or MDField object in each evaluator. The FieldManager does this by calling the postRegistrationSetup() method on each evaluator that is required for the computation.
Note: you can no longer register evaluators or request fields to evaluate once postRegistrationSetup() method is called. \subsection fmd6 D. Setup Worksets If the user plans to use worksets, these objects are typically passed in through the member of the evaluate call as defined in your traits class. In our example, we chose to pass a user defined class called a MyWorkset in to the evaluate call. Since we plan to use worksets, each object of MyWorkset contains information about the workset. Here is the MyWorkset implementation: \code #ifndef PHX_EXAMPLE_MY_WORKSET_HPP #define PHX_EXAMPLE_MY_WORKSET_HPP #include "Phalanx_config.hpp" // for std::vector #include "Cell.hpp" struct MyWorkset { std::size_t local_offset; std::size_t num_cells; std::vector::iterator begin; std::vector::iterator end; }; #endif \endcode The user has written a MyCell class that contains data for each specific local cell. MyWorkset contains iterators to the beginning and end of the chunk of cells for this particular workset. The local_offset is the starting index into the local cell array. The num_cells is the number of cells in the workset. The MyWorkset objects are created one for each workset via the following code: \code // Create Workset information: Cells and EvalData objects std::vector cells(num_local_cells); for (std::size_t i = 0; i < cells.size(); ++i) cells[i].setLocalIndex(i); std::vector worksets; std::vector::iterator cell_it = cells.begin(); std::size_t count = 0; MyWorkset w; w.local_offset = cell_it->localIndex(); w.begin = cell_it; for (; cell_it != cells.end(); ++cell_it) { ++count; std::vector::iterator next = cell_it; ++next; if ( count == workset_size || next == cells.end()) { w.end = next; w.num_cells = count; worksets.push_back(w); count = 0; if (next != cells.end()) { w.local_offset = next->localIndex(); w.begin = next; } } } \endcode Note that you do not have to use the workset idea. You could just pass in workset size equal to the number of local cells on the processor or you could use a workset size of one cell and wrap the evaluate call in a loop over the number of cells. Be aware that this can result in a possible performance hit. \subsection fmd4 E. Call evaluate() Finally, users can call the evlauate routines and the pre/post evaluate routines if required. \code fm.preEvaluate(NULL); // Process all local cells on processor by looping over worksets for (std::size_t i = 0; i < worksets.size(); ++i) { fm.evaluateFields(worksets[i]); // Use workset values . . . } fm.postEvaluate(NULL); \endcode \subsection fmd5 F. Accessing Data Accessing field data is achieved as follows: \code Field< MyVector > ef("Energy_Flux", qp); fm.getFieldData,MyTraits::Residual>(ef); \endcode You do not need to use the Field objects to access field data. You can get the reference counted smart pointer to the data array directly by using the field tag: \code RCP qp = rcp(new FlatLayout("QP", 4)); PHX::FieldTag s("Nonlinear Source", qp); Teuchos::ArrayRCP source_values; fm.getFieldData(s,source_values); \endcode */ /* ************************************************************************ */ /* ************************************************************************ */ /*! \page faq Frequently Asked Questions \section Questions \ref faq1 \ref faq2 \ref faq3 \ref faq4 \ref faq5 \ref faq6 \ref faq7 \ref faq8 \ref faq9 \section faq1 1. Why name it Phalanx? The phalanx was one of the most dominant military formations of the Greek armies in the classical period. It was a strictly ordered formation. The Phalanx software library figures out ordered dependencies of field evaluators. A second more obscure reference relates to the US Navy. The Phalanx software package was designed to provide nonlinear functionality for the Intrepid itegration library. Intrepid was the name of an aircraft carrier during in WW II. Modern US aircraft carriers are protected by a Close-In Weapons System (CIWS) named the Phalanx. Finally, the PI of this project is an avid strategy warfare gamer and leans towards military references. \section faq2 2. Why does the evaluation type exist? This allows for multiple evaluations that used the same Scalar type, but different evaluators specific to the CalculationType. For example, evaluating the Jacobian (derivative of the residual with respect to the degrees of freedom) and the parameter sensitivities (derivative of the residual with respect to the parameters) both use the same Sacado data type. They both want derivatives but with respect to a different set of entities. \section faq3 3. Why do evaluation types have arbitrary data types. Shouldn't all fields of an evaluation type require the same scalar type? In 99% of the use cases, the answer is yes! By changing the scalar type for a specific type of computation, you can break the dependency chain and get incorrect answers. For example, if computing a Jacobian, you would use a FAD scalar type instead of a double scalar type for the evaluation. By mixing FAD and double fields in an evaluation type, you will lose derivative components and get incorrect sensitivities. But there are some use cases where one might want to use different data types. If the user knows that certain evaluation routines have no derivative components, they can bypass the AD chain by switching to doubles for that part of the computation. The is an expert only user mode, but we allow for it. \section faq4 4. Why are the evaluators templated on the evaluation type? Each evaluator must be templated on the evaluation type to be able to (1) allow for the expert use mode described in FAQ \ref faq3; (2) use the automated factory class (PHX::EvaluatorFactory) to build the evaluator for each evalution type. \section faq5 5. Why is the MDALayout object not templated on the ArrayOrder type? This introduces many complications in the comparison operator (operator==) during a comparison of a reverse and natural data layout with the same (but reversed index types). Should these objects be the same? Probably, but this might introduce some confusion. So in an effort to keep things as clean as possible, there is no differentiation based on forward or reverse (Fortran) array ordering. If we were to introduce this, then we would have to add another method to the DataLayout base class to allow for checking the ordering. If mixing natural and reverse ordering becomes common, we may revisit this decision. \section faq6 6. My code seems to be running slow. How can I speed it up? See the section on \ref performance in the Users Guide that gives recomendations on how to maximize the efficiency of your code. \section faq7 7. Compilation take a long time when minor changes are made to an evaluator. How can I speed this up? Explicit template instantiation can be used to speed up evaluator development builds. See the section on \ref performance in the Users Guide. \section faq8 8. Valgrind gives a memory leak error in all evaluators when using the PHX::ContiguousAllocator with Sacado::DFad based fields. What is the problem? The contiguous allocator is allocating space for all variables in an evaluation type in a single contiguous chunk of memory. We allow for multiple scalar types in an evaluation type so the array is allocated using the type char and then based on scalar type sizes (using sizeof method) along with proper alignment, pointers to blocks of memory are handed to the fields using a reinterpret_cast. DFad uses new to allocate the derivative arrays, so this memory is not contiguous. The problem is that the field array memory is deleted as a char instead of it's various data types, so all DFad derivative arrays are leaked during the destruction. There are two ways to alleviate this: (1) write a special Allocator to call the destructor of the field type for each field. (2) Have Sacado implement a version of DFad that allows the user to allocate the DFad array and pass in the pointer during construction. This will also require a new allocator. Choice (2) will be handed in Phalanx in a later release. Our current suggestion is that one should use DFad only if using the PHX::NewAllocator and use SFad if using the PHX::ContiguousAllocator. \section faq9 9. The evaluators contain boilerplate. Is there an easy way to automate creating the evaluators? YES! There is a python script that will automatically generate the skeleton for evaluators. It is found in the "scripts" directory of phalanx and is called "phalanx_create_evaluator.py". It will be install in the "bin" directory when trilinos is installed. Example use can be found at the end of section \ref user_guide_step5 of the \ref user_guide. */ /* ************************************************************************ */ /* ************************************************************************ */ /*! \page developer_notes Developer Notes \section dnotes_notz Description This is a description Pat Notz called his elevator speech on Phalanx: \verbatim The {NAME} package is a library to help decompose a complex problem into a set of simpler problems with managed dependencies. Two benefits of decomposing a problem using {NAME} are (1) increased flexibility because each simpler piece becomes an extension point that can be swapped out with different implementations and (2) easier to craft code because each piece is simpler, more focused and easier to test in isolation. The counter-benefits to these are (1) a potential performance loss due to fragmentation of the over-all algorithm (e.g., several small loops instead of one large loop) and (2) a potential loss of visibility of the original, composite problem (since the code is scattered into multiple places). Managing these trade-offs can result in application code that both performs well and supports rapid development and extensibility. \endverbatim \section dnotes_contiguous_allocator Contiguous Allocator Simple concept for chunk-layout of heterogeneous members with proper data alignment. A 'malloc'ed chunk of memory is always suitable to hold any data type, or array of data types (do a 'man malloc'). \verbatim unsigned char * chunk = malloc( chunk_size ); \endverbatim If a 'double' is to occupy a portion of the chunk then is must be aligned such that: \verbatim double * d = (double *)( chunk + offset_d ); \endverbatim where: \verbatim 0 == offset_d % sizeof(double); \endverbatim The key point is for the offset of each member within the chunk to have the correct alignment, and of course not overlap any other member. \section dnotes_old_description Old email with description of variable manager Email description of Phalanx (formerly known as the variable manager): Phalanx is a tool for controlling the evaluation of data used in the assembly of a residual/Jacobian fill for PDE solvers. I implemented the one in Charon and one also exists in Aria (although it is used for a different purpose). The idea behind the VM is to allow for fast changes to dependency trees. The VM provides variable quantities. A variable consists of a name (string), a type (scalar, vector, tensor - all templated for ad), and the place that it lives (quadrature point, node, edge, face, element, etc...). A variable is evaluated using a provider. The provider contains lists of variables it provides (evlauates) and a list of variables it requires to perform the evaluation (its dependencies). The VM figures out the correct order to call the providers to get the dependency chain correct. I developed this for Charon to address the following issues: 1. Determine the order of variable evaluation so that the correct evaluation is performed: For example, to compute the density, it could be a constant, it could be a function of temperature or it could be a function of temperature and pressure. A factory creates the density provider and from this, the manager automatically figures out the correct evaluation order - whether T and/or P will have to be evaluated before the density. This is an easy example, but there are much more complex dependency chains. 2. Provides a single place to store all evaluation variables so that the procedure by which the variable is evaluated is not known by the objects using the variable. For example, in Charon a quantity of interest is the velocity components. The velocity could be a degree of freedom or it could be a user specified constant, or it could be a user specified field (read from an exodus file). Before we had the variable manager, everywhere in Charon that used velocity, there were repeated loops that looked like: \verbatim if (velocity is DOF) // get values from dof structure else if (velocity is auxiliary data) // go get from aux data structure else if (...) \endverbatim There were hard coded loops in multiple places that had to be updated every time we added a new way to specify velocity. Now, when velocity is required, you just go to the VM and get it. No questions asked. 3. By having a single place to store the variables, we eliminated the reevaluation of temporary quantities that are used in different PDE operators and different equations. Prior to using the variable manager, the reuse of variables like density was ignored. It was recomputed for each operator evaluation. After implementing the variable manager, my residual evaluation times were reduced by a factor of 7 for reacting flow problems! 4. By having the evaluation order, one can perform automatic differentiation without any library like saccado. When the user writes a provider for a function, they can also write a local Jacobian for the individual function. Then by using the chain rule, the local element Jacobian can be assembled by traversing the dependency tree generated by the variable manager. Ariawrote their VM to do this, since Sacado was not around when they started. Aria does use Sacado for user defined functions, but uses the VM for internal functions. There are other benefits, but you get the idea. A primary goal of my current ASC funding is to generalize the variable manager in Charon (there are many things I can do better now to make it more efficient and easier to use) as a trilinos package to be used with Intrepid to aid in element assembly for nonlinear equation sets. */ /* ************************************************************************ */ /* ************************************************************************ */ /*! \page junk Junk \todo Nothing pending! */ /* ************************************************************************ */ /* ************************************************************************ */ #endif