Recent Changes

1. Update your bash file. This file is nearly blank, and not a copy of what was on Scinet. You will need to add the following at the end of the bash.
export sys="niagara"
HOST=$(uname)
ARCH=$(uname)_$(uname -m)
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/$ARCH/lib:$HOME/jetstream/cgnslib_2.5/LINUX
# Load required modules
module load intel/2018.2

For Niagara, execute the following
cd $HOME/petsc-3.8.4
./configure --with-debugging=0 --with-c2html=0 --with-blas-lapack-dir=$MKLROOT
4. Compile PETSc library
After the configuration from the previous step is complete, you will see output similar to below. Execute the first make command.

3. If you are using any of the PetSc functionality, follow the instructions for getting PetSc here. NOTE: This is temporary. Once the issues with the Niagara PetSc module have been worked out, we will use the system module.
Add the environment variables PETSC_DIR and PETSC_ARCH given by the PetSc configure script to your .bash file with, e.g.,
export PETSC_DIR=/gpfs/fs1/home/z/zingg/<username>/petscPETSC_DIR=/gpfs/fs1/home/z/zingg/<username>/petsc-3.8.4
export PETSC_ARCH=arch-linux2-c-opt
4. Turn the petsc flag on/off as appropriate in the Jetstream makefile and compile everything with

cd
git clone -o central <your-oddjob-username>@oddjob.utias.utoronto.ca:/nfs/carv/d1/people/comp-aero/jetstream.git jetstream
PetSc here. Even if you already have PetSc installed, you will have to update it. NOTE: This
Add the environment variables PETSC_DIR and PETSC_ARCH given by the PetSc configure script to your .bash file with, e.g.,
export PETSC_DIR=/gpfs/fs1/home/z/zingg/<username>/petsc

Support has been added to solve the mesh movement and mesh adjoint equations in parallel with CG preconditioned with Additive-Schwarz using PETSc. If you want to use that feature you'll need to download, compile and install PETSc yourself.
Follow the steps below to install PETSc for use with Jetstream:
1. CloneGet version 3.8.4
Download the git repository containing the PETSc library
Gotarball from here and move it to your home directory on SciNet and clone PETScthe HPC system, then unpack it
http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.8.4.tar.gz
cd $HOME
git clone -b maint https://bitbucket.org/petsc/petsc petsc
tar -xvf petsc-3.8.4.tar.gz
2. Roll back PETSc to version 3.8.4 that is compatible with Jetstream
cd $HOME/petsc
git checkout -b e162b4a e162b4a
(Versiou 3.2 with git ID ea79f89 was used previously. The above is for 3.8.4 and is that version currently used.)
3. Configure PETSc system
For SciNet,Graham, execute the following commands:
cd $HOME/petsc
./configure --with-debugging=0 --with-blas-lapack-dir=/scinet/gpc/intel/ics/composer_xe_2011_sp1.9.293/mkl
For Graham or Niagara, execute NOTE: On Graham, you should be using the following commands:Petsc module instead!
cd $HOME/petsc$HOME/petsc-3.8.4
./configure --with-debugging=0 --with-c2html=0 --with-blas-lapack-dir=$MKLROOT
For Niagara, execute the following
cd $HOME/petsc$HOME/petsc-3.8.4
./configure --with-debugging=0 --with-c2html=0 --with-blas-lapack-dir=$MKLROOT
4. Compile PETSc library
xxx=========================================================================xxx
Configure stage complete. Now build PETSc libraries with (cmake build):
make PETSC_DIR=/home/z/zingg/<your_username>/petsc/PETSC_DIR=/home/z/zingg/<your_username>/petsc-3.8.4/ PETSC_ARCH=arch-linux2-c-opt all
or (experimental with python):
PETSC_DIR=/home/z/zingg/<your_username>/petsc/PETSC_DIR=/home/z/zingg/<your_username>/petsc-3.8.4/ PETSC_ARCH=arch-linux2-c-opt ./config/builder.py
xxx=========================================================================xxx
5. Compile Jetstream with PETSc

cd $HOME
git clone -b maint https://bitbucket.org/petsc/petsc petsc
PETSc to a version 3.8.4 that is
cd $HOME/petsc
checkout -b e162b4a e162b4a
(Versiou 3.2 with git ID ea79f89 ea79f89was used previously. The above is for 3.8.4 and is that version currently used.)
3. Configure PETSc to compile on your system
For SciNet, execute the following commands:
cd $HOME/petsc
./configure --with-debugging=0 --with-blas-lapack-dir=/scinet/gpc/intel/ics/composer_xe_2011_sp1.9.293/mkl
For Graham,Graham or Niagara, execute the following commands:
cd $HOME/petsc
./configure --with-debugging=0 --with-c2html=0 --with-blas-lapack-dir=$MKLROOT
For Niagara, execute the following
cd $HOME/petsc
./configure --with-debugging=0 --with-c2html=0 --with-blas-lapack-dir=$MKLROOT

3. If you are using any of the PetSc functionality, follow the instructions for getting PetSc here. Even if you already have PetSc installed, you will have to update it. NOTE: This is temporary. Once the issues with the Niagara PetSc module have been worked out, we will use the system module.
Add the environment variables PETSC_DIR and PETSC_ARCH given by the PetSc configure script to your .bash file with, e.g.,
export PETSC_DIR=/gpfs/fs1/home/z/zingg/<username>/petsc-3.8.4PETSC_DIR=/gpfs/fs1/home/z/zingg/<username>/petsc
export PETSC_ARCH=arch-linux2-c-opt
4. Turn the petsc flag on/off as appropriate in the Jetstream makefile and compile everything with

cd
git clone -o central <your-oddjob-username>@oddjob.utias.utoronto.ca:/nfs/carv/d1/people/comp-aero/jetstream.git jetstream
PetSc here. Even if you already have PetSc installed, you will have to update it. NOTE: This
Add the environment variables PETSC_DIR and PETSC_ARCH given by the PetSc configure script to your .bash file with, e.g.,
export PETSC_DIR=/gpfs/fs1/home/z/zingg/<username>/petsc-3.8.4

Click here to return to top level Code Management page.
The central repository contains the updated baseline version of Jetstream. Have a look at the ‘Merge Notes’ page on the wiki to see what’s changed:
Please follow the instructions below to update your master version of Jetstream.
Clean up your testing branch and commit any pending changes
Switch over to your master branch:
git checkout master
Merge your testing branch back into your master branch
git merge <name_of_your_testing_branch>
Pull the latest and greatest from the central repository:
git pull central master
Delete the testing branch:
git branch -D <name_of_your_testing_branch>
Create a new branch and start working with your updated version of Jestream:
git branch <new_working_branch>
git checkout <new_working_branch>