This file contains information about how I install the optional auxillary
packages.  It is intended to be notes to myself, to remind me how I
previously built the application when I upgrade to a new version, or install
on a new machine.  But you might find some of this information useful.

General approach
----------------

I use an environment variable <app name>_HOME (e.g. ARPACK_HOME) to give
the root directory of the installation of an application.  I install
applications under /local/apps.  In general, the home directory contains lib
and include subdirectories with the libraries and include files (after
installation), but the details depend on the philosophy of the application.

When the application requires a different compilation for different Fortran 90
compilers, I put a separate subdirectory of the home directory for each
compiler, and put library and module files in lib and include subdirectories
under each of them, e.g. $LAM_HOME/lahey/lib and $LAM_HOME/lahey/include.  The
name of the subdirectory is the same as the value for $PHAML_F90 in mkmkfile.sh.

When the application requires a different compilation for different MPI
libraries, I put a separate directory for each MPI library under the compiler
directory (or under the home directory if the compiler directory isn't used),
e.g. $MUMPS_HOME/lahey/mpich/lib.  The name of the subdirectory is the same
as the value for $PHAML_PARLIB in mkmkfile.sh.

Absoft f90
----------

The flags to have Absoft f90 use the single underscore for external names are
-YEXT_NAMES=LCS -YEXT_SFX=_ -YCFRL=1

ARPACK
------

The version I have tested is:
arpack96.tar.gz  Sep 24 1996
patch.tar.gz     Aug  2 2001
parpack96.tar.gz Oct  7 1997
ppatch.tar.gz    Aug  2 2001

cp ARMAKES/ARmake.MPI-SUN4 ARmake.inc

home = <top ARPACK source directory>
PLAT    = LINUX
If LAPACK and/or BLAS libraries are available, change DIRS as instructed.
ARPACKLIB  = $(home)/libarpack.a
PARPACKLIB = $(home)/libparapack.a
FC      = mpif90  (mpif77 for LAM)
FFLAGS  = -O -fno-second-underscore (g77, g95, pathscale) -ff90 (g77)
LAMHF77=ifort (if using lam and desired compiler is not the default)
OMPI_F77=ifort (if using OpenMPI and
OMPI_FC=ifort   desired compiler is not the default)
PFC     = mpif90
PFFLAGS = -O -fno-second-underscore (g77, g95, pathscale) -ff90 (g77)
MAKE = /usr/bin/make

Copy <mpi include directory>/mpif.h to
    ARPACK/PARPACK/SRC/MPI/mpif.h
    ARPACK/PARPACK/UTIL/MPI/mpif.h
    ARPACK/PARPACK/EXAMPLES/MPI/mpif.h

make lib
make plib

The -ff90 flag is because of the use of "real(double precision complex)"
which g77 says is ambiguous as to whether it should return a single or
double precision result.  -ff90 forces a double precision result.

For the NAG compiler include flags -dusty -w=all -dcfuns

ATLAS
-----

On Linux, I build the ATLAS libraries, and merge their LAPACK into the
standard LAPACK library.

make

This will ask several questions; the answers seem pretty obvious.  One of
them will assign an architecture name (e.g. Linux_P4SSE2) which I refer
to as <arch> below.

Then edit Make.<arch>
  On F2CDEFS, remove second underscore from -DAdd__
  Set F77 to the compiler
  On F77FLAGS, add -fno-second-underscore (for g77 et al.)

make install arch=<arch>

Merge the ATLAS partial lapack library into a copy of the system lapack library:
  cd lib/<arch>
  mkdir tmp
  cd tmp
  ar x ../liblapack.a
  cp /usr/lib/liblapack.a ../liblapack.a
  ar r ../liblapack.a *.o
  cd ..
  rm -rf tmp

then move lib/<arch>/lib* to where I want them located.

Because the existing LAPACK library being merged with this was compiled with
g77 (/usr/lib/liblapack.a on Linux), this will require linking in g2c, so
might as well just make one version of ATLAS with g77.

BLACS
-----

This is no longer useful.  It was previously only used for MUMPS.  Now
configure PETSC to include MUMPS, ScaLAPACK and BLACS.

BLACS Version 1.1, patch level 3

mpiblacs.tgz, mpiblacs-patch03.tgz from netlib.

cp BMAKES/Bmake.MPI-LINUX Bmake.inc; make the following changes:
BTOPdir = <top BLACS source directory>
add lib to the library names in BLACSFINIT, BLACSCINIT, BLACSLIB.  I also
  remove the library designations from the names
MPIdir = /usr/lib64/lam
MPILIB =
INTFACE = -DAdd_
TRANSCOMM = -DUseMpi2 (see MPI error #12 in errata.blacs)
F77 = mpif90 (mpif77 for LAM)
F77NO_OPTFLAGS = -fno-second-underscore (for g77, g95, pathscale)
LAMHF77=ifort (if using lam and desired compiler is not the default)
OMPI_F77=ifort (if using OpenMPI and
OMPI_FC=ifort   desired compiler is not the default)

make mpi

To clean up
make mpi what=clean

For MPICH, as above except obvious changes, and
TRANSCOMM = -DUseMpich

DRUM
----

DRUM 20070511

Unpack both Zoltan and DRUM in the apps directory, and set up the config
files for both of them.  Zoltan's config should have DRUM variables at the
end.  DRUM config is in drum/config.  Also check the xml file in
drum/example/xmls.  Set up DRUM with "gmake setup" and then build it
with "gmake".  Build Zoltan after DRUM.

GLUT
----

GLUT 3.7.1

From the distribution that is available on the f90gl web page.  On Linux.

mkmkfiles.imake
make

The only files that are needed are <gluthome>/lib/glut/libglut.a,
<gluthome>/include/GL/glut.h and <gluthome>/include/GL/glutf90.h.
I copy these to the Mesa lib and include directories.

The MesaGLUT distribution also works starting with Mesa version 7.0.3.
See Mesa instructions for this version.

HYPRE
-----

hypre version 1.6.0
              -----

To use a single underscore in name mangling, edit src/utilities/fortran.h
and change the definitions of hypre_NAME_FORT_CALLING_C and friends.

in src:
configure --prefix=$HYPRE_HOME/lahey/lam --with-F77=lf95 --with-mpi-include=$LAM_HOME/lahey/include

also include --with-F77FLAGS=-fno-second-underscore for Gnu related compilers

make
setenv HYPRE_INSTALL_DIR $HYPRE_HOME/lahey/lam
make install

hypre version 1.9.0b
              ------

Example for Lahey f95 and LAM

To use a single underscore in name mangling, edit src/utilities/fortran.h
and change the definitions of hypre_NAME_FORT_CALLING_C and friends.

in src:
./configure --prefix=$HYPRE_HOME/lahey/lam --with-F77=lf95 --with-MPI-include=$LAM_HOME/lahey/include

also include --with-F77FLAGS=-fno-second-underscore for Gnu related compilers
Actually, control of name mangling did not work.  Awaiting reply.

On raritan with openmpi I had to use
--with-MPI-include="$PREFERRED_LIBDIR $PREFERRED_LIBDIR/openmpi"
because mpi.h has #include "ompi/mpi/cxx/mpicxx.h" but ompi is a
subdirectory of openmpi.  Why this never was a problem with other programs
is a mystery to me.

On looney there was some confusion between OpenMPI and LAM.  Avoid it with
./configure --prefix=/local/apps/hypre/nag/openmpi --with-MPI-include="/local/apps/openmpi/nag/include" --with-MPI-libs="mpi orte opal dl" --with-MPI-lib-dirs=/local/apps/openmpi/nag/lib --with-MPI-flags=-pthread CC=gcc F77=g77 F90=nagf95 FFLAGS=-fno-second-underscore

make
make install
chmod o+r $HYPRE_HOME/lahey/lam/include/*

hypre version 2.0.0 and 2.6.0b
              -----     ------

in hypre/src
./configure --prefix=<hypre installation directory> CC=mpicc F77=mpif90 F90=mpif90

In some cases I also needed --without-fei

make
make install

Sometimes it doesn't seem to matter which Fortran compiler compiles it, so
compile it with one of them (e.g. Intel) and then make directory links for
the others, e.g. ln -s /local/apps/hypre/intel /local/apps/hypre/lahey
Then if the tests fail, compile it with other compilers.

LAM
---

On Fedora Core 6, prebuilt LAM is in:
libraries: /usr/lib/lam
include: /usr/include/lam
executables: /usr/bin, some have nonstandard names starting with lam

LAM Version 7.1.1

unpack tar ball in a different place than the install directory

setenv FC <fortran compiler>
setenv FFLAGS -fno-second-underscore (if using g77, g95 or gfortran)
configure <see flags below>
make
make install

switches for configure for a compiler specific subdirectory (e.g. lahey f95):
--prefix=/local/apps/lam/lahey '--with-rsh=ssh -f' --without-mpi2cpp --without-romio

For Intel ifort 11.1.046 (and later?) also need --with-memory-manager=none

Also build it without Fortran to make the non-fortran related commands
(e.g. lamboot) available from $LAM_HOME (which is /local/apps/lam in this case):
--prefix=/local/apps/lam --without-fc '--with-rsh=ssh -f' --without-mpi2cpp --without-romio

mv or cp the doc directory to $LAM_HOME

MESA
----

Mesa Version 7.0.3

If you are also doing MesaGLUT, unpack it on top of Mesa before compiling
Mesa.

You might want to edit <mesa home>/src/mesa/main/config.h to change
DEFAULT_SOFTWARE_DEPTH_BITS to 31 to fix a transparency problem where a
hidden side of a surface comes through the front.

Edit configs/default and change INSTALL_DIR to the installation directory,
e.g. /local/apps/mesa

Edit configs/linux to remove -DHAVE_POSIX_MEMALIGN, because LAM's free() and
posix_memalign() don't play together nicely.

make linux-x86  (on 64 bit arch, use make linux-x86-64)
make install

on 64 bit machines, link lib64 to lib so I don't have to change the location
of the Mesa/glut/f90gl libraries in all my makefiles

On Cygwin, ./configure --prefix=/home/mitchell/apps/mesa --disable-egl --disable-glw --disable-gallium
make
make install
cp lib/*.dll.a to $MESA_HOME/lib
find cygGL-1.dll, cygGLU-1.dll and cygGLUT-3.dll and move them to a directory
  in your executable path, e.g. ~/bin, or make $MESA_HOME/bin in your path

MPICH
-----

I have never had much success with MPICH.  Use MPICH2 unless your system
already has MPICH installed, in which case you don't need hints for
building it.

MPICH2
------

MPICH2 Version 1.0.2p1

setenv F77 lf95
setenv FFLAGS -fno-second-underscore (for g77, g95, gfortran, pathscale)
setenv F90 lf95
setenv F90FLAGS -fno-second-underscore (for g77, g95, gfortran, pathscale)
./configure --prefix=/local/apps/mpich2/lahey |& tee configure.log
For a stand alone machine (for debugging/development) also --with-pm=gforker
  or gforker:mpd
make |& tee make.log
make install |& tee install.log

MUMPS
-----

MUMPS 4.10.0

I no longer need the parallel version; use --download-mumps when building PETSc.
But I still need to build the sequential version because PETSc (at least at
Version 3.3-p6) does not build sequential MUMPS, you need --with-mumps-dir.

For a parallel version:
cp Make.inc/Makefile.INTEL.PAR Makefile.inc (or replace INTEL with appropiate)

Changes to Makefile.inc
CC = mpicc
FC = mpif90 (mpif77 for lam)
FL = mpif90 (mpif77 for lam)
LAMHF77=ifort (if using lam and desired compiler is not the default)
OMPI_F77=ifort (if using OpenMPI and
OMPI_FC=ifort   desired compiler is not the default)
SCALAP = -L/local/apps/scalapack/nag/lam/lib -lscalapack -L/local/apps/blacs/nag/lam/lib -lblacs -lblacsF77init -lblacs -llapack
INCPAR =
LIBPAR = $(SCALAP)
LIBBLAS = -lblas

For a sequential version:
cp Make.inc/Makefile.INTEL.SEQ Makefile.inc (or replace INTEL with appropriate)

FC = <compiler command>  if necessary
FL = <compiler command>  if necessary
LIBBLAS = -lblas
add -fPIC to OPTF, OPTL and OPTC if making PETSc shared libraries

make   (makes the double precision library)
make z (if you also want to make the double complex library)
make alllib (to make s, d, c and z libraries)
move lib/* and copy include/ to the desired locations
for sequential, copy libseq/libmpiseq.a to desired lib

Versions 4.8.4 to 4.9.2 have a bug.  The workaround consists in inserting
"        IF (N .LE. 1) RETURN"
at the beginning of [SDCZ]MUMPS_181.

For g77 related compilers, add -fno-second-underscore to OPTF

For the NAG compiler, need additional flags (add to OPTF)
-kind=byte because they use KIND=8 explicitly
-w=all because they have a lot of unused variables, extensions and obsolescent
-mismatch_all because calls to MPI routines have different types
-dcfuns because of nonstandard double complex intrinsics (for make z)

In Version 4.9, With the NAG compiler I had to move the declaration of IW in subroutine
DMUMPS_227 because it didn't like using LIW as the dimension before it was
declared. (fixed in 4.9.2, 8/19/10)

With Absoft compiler, need -YNO_CDEC

For g95 -Wno=155 supresses warnings

In nag/*/include/dmumps_struc.h, use
integer, parameter :: I8B = selected_int_kind(18)
to replace the 8-byte integer declarations.

Open MPI
--------

On Fedora Core 6, prebuilt Open MPI is in:
libraries: /usr/lib/openmpi
includes: /usr/include/openmpi
executables: /usr/share/openmpi/bin32
The openmpi compilers, like /usr/bin/mpif90, all point to the same name in
/etc/alternatives, which then all point to /usr/bin/opal_wrapper-32.  I have
no idea what that's all about.

Open MPI Version 1.1.2

First build a version without Fortran to get the commands in $OPENMPI_HOME/bin

./configure --prefix=/local/apps/openmpi --disable-mpi-f77 --disable-mpi-f90
make
make install
make clean

Then build it for each Fortran compiler, e.g for Lahey compiler on Linux:

./configure --prefix=/local/apps/openmpi/lahey F77=lf95 FFLAGS=-O FC=lf95 FCFLAGS=-O
make
make install
make clean

Note: The NAGWare f95 compiler needs -mismatch, and -w is also useful.

ompi_info gives information about the installation

run program with:
mpiexec -hostfile <filename> -np <num procs> <executable>
hostfile is not needed for batch systems like PBS.  Otherwise, if hostfile is
omitted all processes run on the local host.
For a master/slave program, use -np 1

ParMETIS
--------

ParMetis Version 3.1.0

On Linux, for use with LAM, in Makefile.in change mpicc to hcc in two places,
and make.

I have also had good luck with setting CC and LD to gcc and INCDIR to the
directory containing mpi.h (for building libraries, but not tests).

Then mkdir lam/lib and move the two libraries to that directory.

May need to add read and execute permission for others on ParMETIS directory.

PETSc
-----

PETSc Version 3.3.6 (Versions earlier than 2.3.3 may require changes in
                     mkmkfile.sh, petsc_init.F90, and petsc_interf.F90.
                     Search them for "before" for the changes.)

setenv PETSC_DIR `pwd`
setenv PETSC_ARCH linux

setenv LAMHF77 ifort (if using lam and desired compiler is not the default)
setenv OMPI_F77 ifort (if using OpenMPI and
setenv OMPI_FC ifort   desired compiler is not the default)

./configure --prefix=<install directory> --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpic++ --with-debugging=0 --with-shared-libraries=0 --with-x=0

make
make install

clean up with:
make allclean

If compiling for use with SLEPc, omit --with-x=0
If debugging is desired, omit --with-debugging=0

To make a sequential version:
   add --with-mpi=0 to the configure flags and change the compilers

NOTE: 10/19/12 when compiling on pepe, do not use the newer gcc that I installed
      to get newer gfortran, because it conflicts with /usr/include/sys/stat.h

To include support for SuperLU, add
  --download-superlu  for sequential
  --download-superlu_dist --download-parmetis --download-metis for parallel
After installing, make a link for libsuperlu_dist.a in the lib directory so I
don't have to change version number in my makefiles every time it changes.

To include support for hypre (parallel only), add
  --download-hypre

PETSc does not support hypre in a sequential build, because of conflicts
between the mpiuni of the two packages, according to a post on petsc-users
4/27/12.

To include support for ML from Trilinos (parallel only), add
  --download-ml

Attempting to include ML in a sequential build gives "Did not find package MPI
needed by ml."

To include support for MUMPS, add

 for sequential
  --with-mumps-dir=/local/apps/mumps/intel/none --with-mumps-serial --with-mpiuni-fortran-binding=0
  Also make sure the MUMPS include directory does not contain mpif.h and mpi.h.

 for parallel
  --download-mumps --download-scalapack --download-blacs --download-parmetis --download-metis

To include support for sequential MUMPS takes more work, prior to Version 3.3.
It was changed in 3.3 as given above, but didn't work for me until patch 4 plus
a special patch they sent me.

  edit <petsc dir>/config/PETSc/packages/MUMPS.py
    change self.liblist to (and omit single precision mumps if necessary)
    self.liblist   = [['libcmumps.a','libdmumps.a','libsmumps.a','libzmumps.a','libmumps_common.a','libpord.a'],
                     ['libcmumps.a','libdmumps.a','libsmumps.a','libzmumps.a','libmumps_common.a','libpord.a','libpthread.a'],
                     ['libcmumps.a','libdmumps.a','libsmumps.a','libzmumps.a','libmumps_common.a','libpord.a','libmpiseq.a']]
    change self.deps to 
    self.deps       = [self.blasLapack]
  use --with-mumps-dir=/local/apps/mumps/intel/none
  use --CFLAGS=-DMPIUNI_AVOID_MPI_NAMESPACE

NAG

  chokes on -pthread and -m64 in mpif90.  See what the compiler flags are in
  /usr/lib64/openmpi/share/openmpi/mpif90-wrapper-data.txt
  and set OMPI_FCFLAGS to them without -pthread -m64.
  If that file doesn't exist, use the flags shown by mpif90 --showme except for
  -pthread, -m64, -I flags, and the libraries.

  NAG gives too many obsolete warnings and needs double complex intrinsics.
  FFLAGS="-w=all -dusty -dcfuns"

  Also, to properly handle iarc and getarg,
  CFLAGS=-DPETSC_HAVE_NAGF90

  I was not able to build PETSc with any Fortran external packages (MUMPS)
  using NAG because of complications with the NAG runtime library.

Absoft
  --FFLAGS=-YNO_CDEC

g95

  sequential, --FFLAGS="-fno-second-underscore -pthread"
  parallel, --FFLAGS=-fno-second-underscore

gfortran

  sequential, with MUMPS, --FFLAGS=-pthread

Lahey

  with a library compiled with Lahey (for example, MUMPS)
  --CC_LINKER_FLAGS="-L/local/apps/lahey/lib64 -lfj90i -lfj90f -lelf"

  for parallel with MUMPS, BLACS needs mpi_init_, and SCALAPACK needs the
  runtime libraries
  --CC_LINKER_FLAGS="-lmpi_f77 -L/local/apps/lahey/lib64 -lfj90i -lfj90f -lelf"

  copy <petsc-src>/linux/*.mod to the include directory

cygwin, gfortran

  make sure cygwin python is installed

  setenv PETSC_DIR `pwd`
  setenv PETSC_ARCH cygwin
  ./configure --prefix=/home/mitchell/apps/petsc/gfortran/none --with-cc=gcc --with-cxx=gcc --with-fc=gfortran --with-windows-graphics=0 --with-mpi=0 --FFLAGS=-fno-second-underscore --download-f-blas-lapack

cygwin/MinGW for a dll

  ./configure --prefix=/home/mitchell/apps/petsc/mingw/none --with-cc=x86_64-w64-mingw32-gcc --with-cxx=x86_64-w64-mingw32-g++ --with-fc=x86_64-w64-mingw32-gfortran --with-windows-graphics=0 --with-mpi=0 --FFLAGS="-fno-second-underscore -static-libgcc -static-libgfortran" --with-ar=x86_64-w64-mingw32-ar --with-ranlib=x86_64-w64-mingw32-ranlib --download-f-blas-lapack

On raritan, I also need
--with-lapack-lib=/usr/lib/liblapack.a --with-blas-lib=/usr/lib/libblas.a
--CC_LINKER_FLAGS=-lpthread

ScaLAPACK
---------

This is no longer useful.  It was previously only used for MUMPS.  Now
configure PETSC to include MUMPS, ScaLAPACK and BLACS.

ScaLAPACK Version 1.8

cp INSTALL/SLmake.LINUX SLmake.inc

changes in SLmake.inc:
home          = <top ScaLAPACK source directory>
BLACSdir      = /local/apps/BLACS/lahey/lam/lib
SMPLIB        =
remove library designations from BLACSFINIT, BLACSCINIT, BLACSLIB (if you did
 that when making BLACS)
F77           = mpif90 (mpif77 for LAM)
CC            = mpicc
NOOPT         = -fno-second-underscore (for g77, g95, gfortran, pathscale)
F77FLAGS      = -O $(NOOPT)
LAMHF77=ifort (if using lam and desired compiler is not the default)
OMPI_F77=ifort (if using OpenMPI and
OMPI_FC=ifort   desired compiler is not the default)
CDEFS         = -DAdd_ -DNO_IEEE $(USEMPI)

make

With the NAG f95 compiler, need -dcfuns -dusty -w=all

SLEPc
-----

SLEPc Version 3.3.3

SLEPc picks up most of its options from the PETSc installation.
Make sure LAMHF77 or OMPI_F77 and OMPI_FC are set to the right compiler.

unsetenv PETSC_ARCH
setenv PETSC_DIR /local/apps/petsc/intel/openmpi
setenv SLEPC_DIR `pwd`
./configure --prefix=/local/apps/slepc/intel/openmpi
  for ARPACK, --with-arpack-dir=/local/apps/arpack/intel/openmpi/lib (NOT!)
  for ARPACK, --with-arpack-flags=-L/local/apps/arpack/intel/openmpi/lib,-larpack,-lparpack
  for BLOPEX, --download-blopex

setenv PETSC_ARCH arch-installed-petsc (before 3.2, installed-petsc)

make              (Or, they tell you some flags to use on these.  Cut and
make install       paste the commands they recommend.)

With Absoft and Lahey, I needed to add a second -larpack after -lparpack

I couldn't get ARPACK to work with nag, gfortran or g95 because of things it
needs in the runtime library.

SuperLU
-------

This is no longer useful.  Instead, configure PETSc to include SuperLU

SuperLU_DIST Version 2.0 of 1/4/06

In make.inc, use the following definitions:
DSuperLUroot    = <top SuperLU source directory>
DSUPERLULIB     = $(DSuperLUroot)/libsuperlu.a
BLASDEF         = -DUSE_VENDOR_BLAS
BLASLIB         = -lblas
LIBS            = $(DSUPERLULIB) $(BLASLIB) $(MPILIB)
CC              = mpicc
CFLAGS          = -O3
NOOPTS          = -O0 $(IMPI)
FORTRAN         = mpif90
FFLAGS          = -O  [-fno-second-underscore for g77 type compilers]
LAMHF77=ifort (if using lam and desired compiler is not the default)
OMPI_F77=ifort (if using OpenMPI and
OMPI_FC=ifort   desired compiler is not the default)
LOADER          = mpif90
LOADOPTS        =
CDEFS           = -DAdd_

make lib

mv libsuperlu.a <superlu lib directory>
cp SRC/*.h <superlu include directory>

TRIANGLE
--------

Triangle Version 1.6

On 32 bit Linux, no changes -- just "make".

On 64 bit Linux, might need to change X11R6/lib to X11R6/lib64

On other systems will probably need to change CSWITCHES in makefile (at
least to remove -DLINUX).

ZOLTAN
------

Zoltan Version 1.52

edit Utilities/Config/Config.nistlinux (or appropriate file) to set
the compiler, MPI library, and ParMETIS location (and other variables
as appropriate).

make YES_FORTRAN=1 ZOLTAN_ARCH=nistlinux zoltan

mv Obj_nistlinux/*.mod <compiler>/<mpi>/include
mv Obj_nistlinux/lib* <compiler>/<mpi>/lib
