             ____  _________________            ______________  ___
            / __ \/ ____/_  __/ ___/_____      / ____/ ____/  |/  /
           / /_/ / __/   / /  \__ \/ ___/_____/ /_  / __/ / /|_/ / 
          / ____/ /___  / /  ___/ / /__/_____/ __/ / /___/ /  / /  
         /_/   /_____/ /_/  /____/\___/     /_/   /_____/_/  /_/   
                                                          
NAME
    PETSc-FEM - a general purpose, parallel, multi-physics FEM program

SYNOPSIS
    This is PETSc-FEM, a general purpose, parallel, multi-physics FEM
    (Finite Element Method) program for CFD (Computational Fluid Dynamics)
    applications based on PETSc . PETSc-FEM comprises both a library that
    allows the user to develop FEM (or FEM-like, i.e. non-structured mesh
    oriented) programs, and a suite of application programs. It is written
    in the C++ language with an OOP (Object Oriented Programming)
    philosophy, keeping in mind the scope of efficiency. PETSc-FEM may run
    in parallel using the MPI standard on a variety of architectures, but
    currently it has been tested in Beowulf clusters only.

    PETSc-FEM is GPL, for further details read the file COPYING in the root
    directory of the distribution.

LICENSE
    This file belongs to the PETSc - FEM package a library and application
    suite oriented to the Finite Element Method based on PETSc. Copyright
    (C) 1999-2013, Mario Alberto Storti, Norberto Marcelo Nigro, Rodrigo
    Rafael Paz, Lisandro Dalcin and Ezequiel Lopez, Centro Internacional de
    Metodos Numericos en Ingenieria (CIMEC-Argentina), Universidad Nacional
    del Litoral (UNL-Argentina), Consejo Nacional de Investigaciones
    Cientificas y Tecnicas (CONICET-Argentina).

    This program is free software; you can redistribute it and/or modify it
    under the terms of the GNU General Public License as published by the
    Free Software Foundation; either version 2 of the License, or (at your
    option) any later version.

    This program is distributed in the hope that it will be useful, but
    WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
    Public License for more details.

    You should have received a copy of the GNU General Public License along
    with this program; if not, write to the Free Software Foundation, Inc.,
    59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.

INSTALLATION
  Platforms
    Currently, PETSc-FEM is known to run only on Beowulf class clusters
    based on Intel processors, running the GNU/Linux OS. Off course, you can
    also install PETSc-FEM in a single GNU/Linux box, for use in sequential
    (i.e. non-parallel) mode or either for simulating parallel runs, by
    launching several processes in the same processor. (This is a feature of
    MPI). PETSc-FEM compiles cleanly with the GNU compiler GCC 4.1.1 that
    comes with the Fedora Core 5 GNU/Linux distribution.
    <http://fedora.redhat.com/>

    I guess it should be relatively easy to install in other processor
    architectures (SPARC or Alpha) running GNU/Linux and also on other Unix
    flavor's.

  Required Software
    Before installing PETSc-FEM, please verify you have the following
    packages correctly installed in your system.

    MPI *Message Passing Interface*. We are currently using MPICH2 1.0.3
        (<http://www.mcs.anl.gov/mpich>) but I think it should be possible
        to use other versions of MPI like LAM or OpenMPI.

    PETSC
        *Portable, Extensible Toolkit for Scientific Computation* and can be
        found at <http://www.mcs.anl.gov/petsc/>. Current version of
        PETSc-FEM compiles with PETSc 2.1.6.

    LAPACK
        This is, in fact, a prerequisite to install PETSc and can be found
        at <http://www.netlib.org/lapack/>.

    Libretto
        This is a C library of generic containers and can be found at
        <http://pobox.com/~aaronc/tech/libretto/>. I have learned recently
        that development of Libretto has been stopped since it has been
        superseded by GLIB, so that I think that in a future I will replace
        the Libretto code by GLIB. Also, the STL library (that comes with
        gcc) can replace all or some of the features provided by Libretto
        but I found that the STL implementation (at least that one that
        comes with gcc) is rather inefficient.

    GLIB
        GLIB is a library which includes support routines for C such as
        lists, trees, hashes, memory allocation, and many other things. The
        official web site is <http://www.gtk.org/>. [RedHat RPM packages:
        "GLIB", "glib-devel"]

    Newmat
        This is a matrix library to be used mainly at the element routine
        level, i.e. for small matrices and can be found at
        <http://webnz.com/robert/>. It is not mandatory to use Newmat for
        all element calculations. In fact there is included a matrix class
        called FastMat2 that is more performing, but Newmat is still used in
        some internal parts and has much more functions, so that you have to
        install it. Also, you may prefer *Newmat* over *FastMat2* for
        element calculations since it supports overloading of algebraic
        operations. See the documentation on *FastMat2* in the *PETSc-FEM*
        documentation for reasons why to use FastMat2 instead of Newmat.

    METIS
        METIS is a software package for partitioning unstructured graphs,
        partitioning meshes, and computing fill-reducing orderings of sparse
        matrices. It can be found at <http://www.cs.umn.edu/~metis>.

    Meschach
        The Meschach Library is a numerical library of C routines for
        performing calculations on matrices and vectors. It is used only for
        some functions that can not be done with Newmat. Currently, it is
        used in the advective module for performing eigenvalue decomposition
        of non-symmetric matrices. I think that in a future this
        functionality could be replaced by calls to LAPACK.

    Perl
        If you want to use ePerl then you need to have Perl installed.
        [RedHat RPM package: "perl"].

    SuperLU (optional)
        A collection of ANSI C routines to solve sparse linear systems of
        equations. Can be found at <http://www.nersc.gov/~xiaoye/SuperLU/>.
        This package is optional, install it if you think that SuperLU is
        faster than PETSc for direct factorization of sparse matrices.

    ANN (optional)
        This is the *Approximate Nearest Neighbor Library* and can be found
        at <http://www.cs.umd.edu/~mount/ANN>. Currently it is only used in
        the Navier-Stokes module for calculating the damping functions in
        the Smagorinsky model. However, in a future it may be used more
        widely for some mesh manipulations. Currently it is optional.

    pthreads library (optional)
        This is a POSIX compliant library for threads, currently used for
        asynchronous communication with the IBM Data Explorer visualization
        tool, but it might be used for other things in a near future. It
        comes with gcc, so that if you use gcc, you probably have it already
        installed.

    Simple Sockets Library (SSL) (optional)
        The Simple Sockets Library (SSL) allows C programmers to use
        interprocess communications via Berkeley sockets simply and quickly.
        Most of the SSL's functions resemble C's file i/o functions, so C
        programmers will find the SSL easy to learn. Currently used for
        communication with the IBM Data Explorer visualization tool, but but
        it might be used for other things in a near future. [Current
        version: 2.11a Tue Mar 6 09:46:16 EST 2001] [Note: If you do a web
        search you can find it as a file "COSMIC.tar.gz" or "ssl.tar.gz"].

  Unpacking
    Once you got the "petscfem-xxx.tar.gz" package ("xxx" stands for the
    version), gunzip and untar it like this:

      $ gunzip petscfem-xxx.tar.gz
      $ tar xvf petscfem-xxx.tar

    This will create a *petscfem-xxx* directory with all the source files
    included.

  Makefile variables
    "$(PETSCFEM_DIR)"
        The root of the PETSc-FEM package

    "$(PETSC_ARCH)", "$(PETSC_DIR)"
        These are variables relative to PETSc. See documentation.

  Configuring
    Once you have all the software mentioned in "Required Software"
    installed, you have to define the variables in the CONFIGURATION
    VARIABLES section in file $(PETSCFEM_DIR)/Makefile.base.

    *   First you have to set the "PETSC_BOPT" variable which is the default
        value for the "PETSc" "BOPT" variable, which sets the optimization
        level to be used in compilation. You can set it to

        "PETSC_BOPT = g_c++"
            Compiles for debugging.

        "PETSC_BOPT = O_c++"
            Compiles with optimization enabled.

        The optimization level is inherited by the "PETSc" libraries so that
        if you compile with "PETSC_BOPT=O_c++" then not only "PETSc-FEM"
        will be compiled with optimization, but also the "PETSc" optimized
        libraries will be used. The specific optimization options are set in
        "$(PETSC_DIR)/bmake/$(PETSC_ARCH)/variables" ("OCXX_COPTFLAGS"
        variable). According to our experience maximum optimization is
        achieved with <BOPT=-O2 -funroll-loops>.

        The default value can be superseded by, for instance, issuing

          $ make BOPT=O_c++ ns

        This compiles the Navier-Stokes module with optimization enabled
        independently of the "PETSC_BOPT" default value.

    *   The "PROF_FLAGS" variables allows you to turn on profiling. By
        default, profiling is deactivated.

    *   Other configuration variables are defined in a file "Makefile.defs",
        but normally they are set only when you first install PETSc-FEM, so
        the best is to put "Makefile.defs" in the directory immediately
        above PETSc-FEM directory (i.e. "$(PETSCFEM_DIR)/.."). So that, if
        you install another version of PETSc-FEM, the settings will be seen
        immediately by the new version. A version of "Makefile.defs" is
        included in the PETSc-FEM directory. Copy it to the directory
        _above_ the PETSC-FEM directory and configure the variables inside.
        If you install another version of PETSc-FEM then probably you have
        not to configure again these variables, unless a new package has
        been added or you have moved it.

  Compiling
    "cd" to directory "$(PETSCFEM_DIR)" and do

      $ make depend
      $ make libpetscfem        # to build the library
      $ make all                # to build the library and all the modules

    also

      $ make what

    can help you on other useful targets.

    If nothing goes wrong you will end with a src/libpetscfem.a library and
    application binaries (ending in ".bin") in their respective directories,
    e.g. applications/ns/ns.bin is the Navier-Stokes binary.

DOCUMENTATION
    Documentation for *PETSc-FEM* is included in the "$(PETSCFEM_DIR)/doc"
    directory. There is basically a reference manual written in *LaTeX*
    (doc/petscfem.tex), and embedded documentation for the routines to be
    used with "Doc++". In the standard distrib you should find both of them
    converted to HTML ("LaTeX" is converted with "latex2html") and
    accessible from doc/index.html. Other formats (perhaps PostScript and
    PDF) may be also found in the "doc" directory.

TESTS AND EXAMPLES
    Many tests and examples are in the "$(PETSCFEM_DIR)/test" directory.

WRITING A NEW APPLICATION MODULE
    The best way to start coding an application program is to duplicate a
    directory (for instance "applications/advdif") or (for instance
    "applications/ns") to a directory from your own (i.e. outside the
    "$(PETSCFEM_DIR)" tree, say "/your/appl/path". Configure the
    "$(PETSCFEM_DIR)" variable in the local makefile (i.e.
    "/your/appl/path/Makefile").You should have to configure other variables
    as well, perhaps "$(MYOBJS)" (this is a list of the object (".o") files
    in this local directory) and "$(PROG)" (the name of the program to be
    built, without the suffix ".bin").

AUTHORS
    Mario A. Storti* "<mario.storti@gmail.com>"
        PETSc-FEM kernel, NS and AdvDif modules.

    Norberto M. Nigro* "<nnigro@intec.unl.edu.ar""
        NS and AdvDif modules, multi-phase flow.

    Rodrigo R. Paz* "<rodrigop@intec.unl.edu.ar>"
        AdvDif module, hydrology module, compressible flow, fluid-structure
        interaction, preconditioners.

    Lisandro Dalcin* "<dalcinl@intec.unl.edu.ar>"
        PETSc-FEM kernel, Python extension language project, linear algebra,
        preconditioners, multigrid.

    Ezequiel Lopez* "<ezequiellopez@hotmail.com>"
        Mesh relocation algorithms.

    Laura Battaglia* "<lbattaglia@santafe-conicet.gov.ar>"
        Free surface algorithms.

    Gustavo Rios Rodriguez * "<gusadrr@yahoo.com.ar>"
        Adaptive refinement.

    * CIMEC, Internacional Center for Computational Methods in Engineering,
    Santa Fe, Argentina. <http://www.cimec.org.ar/petscfem> Ordinary mail:
    Mario Storti, CIMEC-INTEC, Guemes 3450, 3000 Santa Fe, Argentina.

