BASIC COMMANDS TO BUILD AN EXECUTABLE ON THE LINUX MACHINE
Gnu compilers available on the linux machines
  • gcc (for c and c++)
  • f77  (for f77)
Performance compilers available on the linux machine by Intel
  • ifc ( f95 compiler)
  • icc (c and c++, not yet available)
Debuggers available
  • gdb
  • idb ( main debugger for ifc compiler)

ifc compiler v71 manuals


Version 71
 Html form of Manual
 html
Version 71
 error listing at compile and run time
 pdf
Version 71
 fortran libraries
 pdf
Version 71
Intel Fortran Reference Manual
 pdf
Version 71
 Intel Compiler Quick Reference Manual
 pdf
Version  71
Debugger
html
Matlab Engine
Using the Matlab engine for animation html
Fortran90 page, html form by Dr Clive Page
Fortran90 in pdf form
CTC manual on f90 and f77
f77 manual ( You have to know some f77 !)
nsc manual on f90 Main Page by Bo Einarsson and Yurij Shokin
Michael Metcalf f90 manual
Boston University F90 Manual too good to miss !!

DIRECTORY STRUCTURE FOR ALL PROJECTS

  • create a directory called src for all source files
    • all source files and the Makefile.mk is places in the src/ directory
  • create a directory called bin for the executable
    • the executable that runs the code is placed in the bin/ directory
  • create a directory called dat for all data files
    • Input and output data files are places in the dat/ directory
  • create a directory called report for the report
    • the report regarding the project is places in the report directory
All projects in aero 361 need to follow a strict layout. The purpose of the layout is to organize the work and mitigate clutter when the projects become big --- specially when reports are involved. The reports will be written in LaTex which is basically writing code! The projects will also emphasize post processing from using Matlab, Tecplot and Xmgrace. Thus a single project will consist of source codes in f90, latex source codes, data from the f90 code and tecplot or matlab figures. All these various items will have to be placed in separate directories so that clutter and confusion is minimized and efficiency is maximized: your first lesson in OPTIMIZATION. The figure below illustrates the directory structure for proj1. All other projects will follow the same procedure and layout.

/home/joeusr
|
|
V
aero361/
|
V
proj[name]
|
|
src/     dat/       bin/  report/

directory type

what should be in it

how to name and organize

src/
 source file of the code and the Makefile
  • .f90 files
  • Makefile.mk file
  • Environment scripts for Matlab animation
 Common source code file names:
The following files are a basic minimum list of files that will be common on almost all your big codes. The names should be close to what the purpose of the source code is supposed to be. Calling a routine firefox.f90 when the source code file outputs your results is not correct and will be confusing to Me!!!!!
Names
 Purpose
main.f90
 main program file
input.f90
 input subroutine
driver or solver.f90 main solver routine
output.f90
 tecplot post processing subroutine
global.ff90
 global module file
Makefile.mk
 The makefile file

dat/
 Output and Input data:
  • .plt or .dat files for tecplot
  • .lay and .lpk tecplot files
Data files should also be named appropriately and with the correct suffix. The tecplot or xmgrace data files the code outputs should  have the following choice of suffixes
  • .dat
  • .plt
The tecplot layouts will have .lay suffix
bin/
 executable for running the code
  • code.x
 All executables will be named code.x with the .x suffix. The executables will remain in the bin/ sub-directory. The executable is often stripped or removed and by segregating it from the source codes the chances of removing the source code files is minimized.
report/
 Latex report files
  • .tex files
  • .ps files
  • .fig files

The approach to the latex files is similar to that of the f90 source codes. The entire report is broken down into a preamble file and its subsequent chapter files, hers is an example of some common files
Names
 Purpose
abstract.tex
 abstract of the project
main.tex
 main preamble that calls all other files
introduction.tex
 introduction chapter
theory.tex formulation chapter
results.tex
 results chapter
conclusion.tex
 conclusion chapter
bibliography.tex
 bibliography
appendixa.tex
 appendix a
appendixb.tex
 appendix b


BUILDING A CODE

The building of an executable from a set of source (main.f90, input.f90 solver.f90 and output.f90)  files contains two steps:
  1. compilation step
  2. link step to create the final executable

COMPILE step

The compilation step is performed as follows from a shell prompt

<bash>ifc -c -w -g -r8 main.f90 input.f90 solver.f90 output.f90
where
  • main.f90 is the main program file
  • input.f90 is a subroutine or function file
  • solver.f90 is a subroutine or function file
  • output.f90 is a subroutine of function file
and
  • -c is the compile flag
  • -w is the do not list the warnings file
  • -g is the debugging flag. If you are going to use the debugger then this flag needs to be present in the compile and link step
  • -r8 force double precision in case user has forgotten to do so in the code

Note

  • Every code must have a main program and there can be only one main program per program executable. you cannot create an executable from a set of subroutines only. You have to link all the subprogram files to a main program file. A main program file always contains the header
  • f77 codes are to be saved with a .f suffix. They can be compiled using the ifc compiler or the g77 compiler.
    • program <user defined name>
  • If the project or a set of object files is dependent on a module file then the module file will precede all other files dependent to it in the compile and link step e.g.
    • global.f90 is a module that contains all the global variables and arrays that subroutine driver.f90 needs. In the compile and link step the global.f90 and global.o  will precede subroutine driver.f90 and driver.o or else an error will occur in the compile step.

LINK step

link the f90 object files (.o) and create the executable. When compilation of all source files is successful there will exist files with .o suffix. These object files are linked to create the executable

<bash>ifc -g main.o input.o solver.o output.o -o ../bin/code.x
where
code.x is the executable that is created in the bin dir by linking the above object files.

COMPILE AND LINK IN ONE STEP

One need not separate the compile and link step, they can be combined into one step, but no object file output will exist in the directory. This step is not recommended when using Makefiles.

<bash>ifc -w -g -r8 main.f90 input.f90 solver.f90 output.f90 -o ../bin/code.x

ADDRESS ERROR CHECKING, TYPE DECLARATION MIS-MATCH

If you are getting address error ****  or ????? then compile using the -C flag.  The -C flag performs run time checks so when the code executes it will tell the user in which routine the code bombed. The -C flag should be removed when the code has been debugged otherwise the execution is slow when the code is big. Do a "man" on ifc to see the details of the flag

<bash> man ifc

A common error is when the elements in the argument list between the procedure and the calling program differ in declaration type. This type of error will not be caught by the compiler unless an explicit interface is used. At run time the answers will have a "??????????" associated with output. If there is no explicit interface then the use of the "-C" flag along with the other flags will catch the line where there is a type conformance error in the code at run time.

An address error is common when using arrays and passing them through the argument list. To check for address error or run time errors the -C flag can be invoked. This flag is only to be used for testing the code. When the code is fully tested the code should be re-compiled without this flag. The compile step is shown below. This step needs to be followed by the link step

 <bash>ifc -c -C -w -g main.f90 input.f90 solver.f90 output.f90
<bash>ifc -g main.o input.o solver.o output.o -o ../bin/code.x
or

<bash>ifc -C -w -g main.f90 input.f90 solver.f90 output.f90 -o ../bin/code.x

ATTENTION

When compiling source files that are form repositories or from the Gams library avoid using the -C or -g flag. If the code from the repository is functional but sloppy it could cause problems if the run time error flags are used. Confine the use of the run time error flags to source files written by you.

HOW TO HANDLE MODULE FILES

f90/95 uses modules to perform the function of common files and much more. A module is compiled and linked like any other f90 source file, except it is more powerful and versatile. Procedures can be included in a module and global variables and arrays can be accessed through a module. An explicit interface is a procedure that resides inside a module. When accessing an explicit interface from any external procedure (subroutine, main program, function and module) a USE statement is used to access the module containing the procedure. Global variables/arrays/characters that reside in a module also require the USE statement for access. The syntax and mechanics on Modules can be found here
Modules and Interfaces
    6.1  Example defining constants
    6.2  Module Procedures
    6.3  Public and Private accessibility
    6.4  Avoiding name clashes
    6.5  Pros and Cons of Modules
    6.6  Explicit Interfaces

When using a module e.g. global.f90, which main.f90 and solver.f90 access the compile and link step should be performed as follows

Compile and Link step separate:

<bash> ifc -c -w global.f90 main.f90 solver.f90 output.f90 input.f90
<bash> ifc global.o main.o solver.o output.fo input.o -o ../bin/code.x

Compile and Link step together:

<bash> ifc -w -g global.f90 main.f90 solver.f90 output.f90 input.f90 -o ../bin/code.x

Since main.f90 and solver.f90 depend on items that are in global.f90 they need to follow global.f90 in the compile and link step.

RUNNING A CODE

To run the code locate the executable and type the following lines in the shell. In this example the executable created by linking the object files is "code.x".
  • <bash>./code.x
ALL executables or scripts in the unix shell can only be executed if the "./" precedes the name of the script or executable. The "./" tells the shell that it is to fork a process from the shell that is not listed in the PATH variable. If you do not want to type "./" every time you need to run something then you can add the following line in you .bashrc file. The .bashrc file is a startup file which resides in your $HOME directory
  • <bash> ls -al | grep .bashrc
  • <bash> vi .bashrc
Insert the following line in your . bashrc as shown below ( exactly !!!!). All text start at column 1.
# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi
# PATH VARIABLES ::
#setting up PATH so that you can run executable without the ./name.x command
export PATH=$PATH:.
Bring up a new shell and then Run the code.x executable
  • <bash>code.x

Debug the code

  • command line form
    • <bash>idb code.x
  • gui mode (not available at present)
    • <bash>idb -gui code.x

LINKING TO GNU LIBS AND CODES

Linking to a library in some remote directory or to some gcc or g77 code
In order for and f90 or c++ code compiled using the intel compilers to link to some gcc or g77 compiled code the user needs to invoke the lg2c library routines as shown below.
  • Need to access the library only in the link step
  • -L/ where the library resides
  • -lname of library: replace lib with "l" and omit ".a" suffix
  • example
    • <bash>ifc file.o -L/usr/lib/gcc-lib/i386-redhat-linux/3.2.2 -lg2c -o ../bin/code.x
    • <bash>ifc file1.o file2.o file3.o -L/path/  -lname  -o ../bin code.x

  • lname is the name of the of library
NOTE: In case you you have a newer kernel of are unsure of the gcc version then you need to check it
    • <bash> ls /usr/lib/gcc-lib/i386-redhat-linux <TAB> <TAB>
NOTE:For the current kernel the lib is located at
  • /usr/lib/gcc-lib/i386-redhat-linux/3.2.3/

By pressing the <TAB> key twice the shell will automatically fill the path with the correct version number e.g 3.2.2

RUNNING CODES ON REMOTE MACHINES

BACKGROUND

Optimization  codes run for a long time and waiting for them while logged in is a waste of time. The idea is to submit the code and log out and come back when its done.  Job submission requires you to be responsible and not abuse this service: sending 20 jobs is not a good idea and this service can be discontinued if a user abuses it. You need to remember that others are using these machines too!
  • If you are writing data to the screen, but do not care much for it then you should "standard output"  it into the /dev/null filter so that it does not take up space.
  • If the screen output is required then you need to standard output the file to a file. For iterative schemes this file can fill up and exceed your quota and then you will not be able to log in. Take the following necessary steps:
    • send the data to the file which resides in either
      • /remote/scratch/joesuser
      • /remote/aere/joeuser
  1. <bash>ssh -l username mercury.aere.iastate.edu
  2. <bash>cd into the dir containing the executable

The "nice" option

<OPTIONAL, NOT REQUIRED for whopper and aere machines>
Using the nice command: ADJUST is 10 by default.
Range for linux machines
  • goes from -20 (highest priority) to 19 (lowest).
  • the default is set to 10 so that others can do work on the machine while your "hog" of a code is rampaging through the system.

Optimization flags to Compile the code with:

Always compile and link the production code without the -g option :
Use the -O2 or -O3 flags for speed up. Use -O2 and check to see if the answer is good.  I recommend using -O2 flag and not -O3 flag. Severe optimization can affect the results. Never run a large production code with the -g flag: it will be slow.

Running the code

SENDING DATA TO OBLIVION

  • <bash>nohup nice -10 code.x > /dev/null &
logout from the machine and then log back in and use the "top" command to see if code.x is using the cpu resources
  • <bash> top

SENDING OUTPUT TO A FILE

  • <bash>nohup nice -10 code.x > /remote/scratch/joeuser/output.dat &
  • <bash>nohup nice -10 code.x > /remote/aere/joeuser/output.dat &
NOTE: The symbol "&" sets the process into the background else the process will terminate when the user logs out of the remote machine

Auto Parallelization of loops

The intel compiler can auto parallelize nested loops when it detects them in the code. In order to successfully auto-parallelize the loop from the compiler the loops need to be of stride one. In f90 the arrays are stored in column format to maintain contiguity.  The following conditions must be met to successfully use the auto parallel feature of the compiler
  1. Maintain stride 1
  2. Avoid race conditions
  3. Avoid goto and exit statements in the loop
  4. Avoid calling procedures in the loops
  • <bash> ifc -c -parallel main.f90
NEAT TRICKS FOR BETTER PROJECT MANAGEMENT IN F90
The namelist feature is an extremely powerful and versatile input/output feature in fortran programming. Many large programs that have a lot of input parameters, settings and switches use the namelist feature. It allows the programmer to organize the input file and avoid common mistakes. The example below shows how the file works. The file is like any input file, you do not compile it, only access it through the open statemet.
Input from a file and output to a file using namelist
LINEAR SOLVER PROJECT
The official Linear Solver routine for this class is decomp.f90: this routines has been tested in Gams and contains two procedures (decomp and solve).  There is a driver.f90 program that calls decomp and solve.
The decomp and solve procedures should be in a Module creating and Explicit interface. You are not to modify or add anything into the routines! The contents of the code within each routine must be untouched. You are only to add the Module Interface as shown below.
Module linear_solver_module
implicit none
contains
subroutine decomp(.......)
:
end subroutine decomp
Subroutine solve(.......)
:
end subroutine solve
end module linear solver_module

THE REQUIRED ROUTINES
You will need to compile the linsolver.f90 file 1st since it is now a Module and the only way the driver can access it is through a
  • Use statement
  • Call statement

Removing object files

When removing object files in the src/ directory care must be taken. Inadvertently removing the source files is a common accident among coders. To avoid this discomfort follow the steps below
  1. cd into the src dir
  2. use the command rm -i *.o at the prompt. The -i flag will prompt the user with a yes/no question which is safer.

ERRORS WHILE COMPILING AND LINKING

Any programming project will be fraught will compilation and link errors. Even the consummate practitioner of CFD will experience problems while compiling or linking codes. The problem is not how to fix the errors, but how to develop the necessary skills to ascertain how and where to look for the fix. I have compiled a list of common errors programmers experience and their possible remedies. Not all problems can be accounted for, but the  idea is to gather enough understanding that for every complicated problem there lies a simple fix!