Resources

Python Conventions

Command-line Script Help Message Syntax

Python scripts use the argparse standard library module to parse command-line arguments. This generates help messages that follow a fairly standard syntax. Unfortunately there is no accepted standard, but some style guides are available.

The help message consists of the following sections:

A “usage” line
A quick description of the script, possibly with an example invocation.
Positional and Keyword argument descriptions
Any extra information that would be useful to the user.

The Usage Line

Contains a short description of the invocation syntax.

Starts with usage: <script_name> where <script_name> is the name of the script being invoked.
Then keyword arguments are listed, followed by positional arguments (sometimes these are reversed).
- There are two types of keyword arguments, short and long, usually to save space only the short name is in the usage line.
  - short denoted by a single dash followed by a single letter, e.g., -h.
  - long denoted by two dashes followed by a string, e.g., --help.
Optional arguments are denoted by square brackets [].
A set of choices of arguments that are mutually exclusive are separated by a pipe |
A grouping of arguments (e.g., for required arguments that are mutually exclusive) is done with round brackets ()
A keyword argument that requires a value will have one of the following after it:
- An uppercase name that denotes the kind of value e.g., NAME
- A data type that denotes the value e.g., float or int or string
- A set of choices of literal values (one of which must be chosen) is denoted by curly brackets {}, e.g., {1,2,3} or {choice1,choice2}.

Example: usage: deconvolve.py [-h] [-o OUTPUT_PATH] [--plot] [--deconv_method {clean_modified,lucy_richardson}] obs_fits_spec psf_fits_spec

The script file is deconvolve.py
The -h -o --plot --deconv_method keyword arguments are optional
The --deconv_method argument accepts one of the values clean_modified or lucy_richardson
The keyword arguments are obs_fits_spec and psf_fits_spec

Example: usage: spectral_rebin.py [-h] [-o OUTPUT_PATH] [--rebin_operation {sum,mean,mean_err}] [--rebin_preset {spex} | --rebin_params float float] fits_spec

The script file is spectral_rebin.py
The all of the keyword arguments -h -o --rebin_operation --rebin_preset -rebin_params are optional
The single positional argument is fits_spec
The --rebin_operation argument has a choice of values sum,mean,mean_err
The --rebin_preset and --rebin_params arguments are mutually exclusive, only one of them can be passed
The --rebin_preset argument only accepts one value spex
The --rebin_params argument accepts two values that should be floats

The Quick Description

The goal of the quick description is to summarise the intent of the program/script, and possibly give some guidance on how to invoke it.

Argument Descriptions

Usually these are grouped into two sections positional arguments and options (personally I would call them keyword arguments, as they can sometimes be required) but they follow the same syntax.

The argument name, or names if it has more than one.
Any parameters to the argument (for keyword arguments).
A description of the argument, and ideally the default value of the argument.

Example:

--rebin_operation {sum,mean,mean_err}
						Operation to perform when binning.

Argument is a keyword argument (starts with --)
Argument name is ‘rebin_operation’
Accepts one of sum,mean,mean_err
Description is Operation to perform when binning.

A full argument description looks like this:

positional arguments:
  obs_fits_spec         The observation's (i.e., science target) FITS SPECIFIER, see the end of the help message for more information
  psf_fits_spec         The psf's (i.e., calibration target) FITS SPECIFIER, see the end of the help message for more information

options:
  -h, --help            show this help message and exit
  -o OUTPUT_PATH, --output_path OUTPUT_PATH
						Output fits file path. By default is same as the `fits_spec` path with "_deconv" appended to the filename (default: None)
  --plot                If present will show progress plots of the deconvolution (default: False)
  --deconv_method {clean_modified,lucy_richardson}
						Which method to use for deconvolution. For more information, pass the deconvolution method and the "--info" argument. (default: clean_modified)

Extra Information

Information listed at the end is usually clarification about the formatting of string-based arguments and/or any other information that would be required to use the script/program. For example, in the above fill argument description example the FITS SPECIFIER format information is added to the end as extra information.

Language Syntax Conventions

For those unfamiliar, a quick guide to important bits of python syntax that the command-line interface relies upon. For more information, see the offical Python tutorial.

Python tuple syntax

Tuples are ordered collections of hetrogeneous items. They are denoted by separating each element with a comma and enclosing the whole thing in round brackets. Tuples can be nested.

Examples:

(1,2,3)
('cat', 7, 'dog', 9)
(5.55, ('x', 'y', 0), 888, 'abc', 'def')

Python slice syntax

When specifying subsets of datacubes, it is useful to be able to select a N-square (i.e., square, cube, tesseract) region to operate upon to reduce data volume and therefore processing time. Python and numpy’s slicing syntax is a nice way to represent these operations. A quick explanation of the syntax follows.

Important Points

Python arrays are zero-indexed. I.e., the first element of an array has the index 0.
Negative indices count backwards from the end of the array. If you have N entries in an array, -1 becomes N-1, so the slice 0:-1 selects the whole array except the last element.
Python slices have the format start:stop:step when they are used to index arrays via square brackets, e.g., a[5:25:2] returns a slice of the object a. Slices can also be defined by the slice object via slice(start,stop,step), e.g. a[slice(5,25,2)] returns the same slice of object a as the last example.
The start, stop, and step parameters generate indices of an array by iteratively adding step to start until the value is equal to or greater than stop. I.e., selected_indices = start + step * i where i = {0, 1, ..., N-1}, N = CEILING[(stop - start) / step].
Mathematically, a slice specifies a half-open interval, the step of a slice selects every step entry of that interval. I.e., they include their start point but not their end point, and only select every step elements of the interval. E.g., 5:25:2 selects the elements at the indices {5,7,9,11,13,15,17,19,21,23}
By default start is zero, stop is the number of elements in the array, and step is one.
Only the first colon in a slice is required to define it. The slice : is equivalent to the slice 0:N:1, where N is the number of elements in the object being sliced. E.g., a[:] selects all the elements of a.
Negative parameters to start and stop work the same way as negative indices. Negative values to step reverse the default values of start and stop, but otherwise work in the same way as positive values.
A slice never extends beyond the beginning or end of an array. I.e., even though negative numbers are valid parameters, a slice that would result in an index before its start will be empty (i.e., select no elements of the array). E.g., If we have an array with 5 entries, the slice 1:3:-1 does not select {1, 0, 4}, it is empty.

Details

Let a be a 1 dimensional array, such that a = np.array([10,11,15,16,19,20]), selecting an element of the array is done via square brackets a[1] is the 1^th element, and as python is 0-indexed is equal to 11 for our example array.

Slicing is also done via square brackets, instead of a number (that would select and element), we pass a slice. Slices are defined via the format <start>:<stop>:<step>. Where <start> is the first index to include in the slice, <stop> is the first index to not include in the slice, and <step> is what to add to the previously included index to get the next included index.

E.g.,

2:5:1 includes the indices 2,3,4 in the slice. So a[2:5:1] would select 15,16,19.
0:4:2 includes the indices 0,2 in the slice. So a[0:4:2] would select 10,15.

Everything in a slice is optional except the first colon. The defaults of everything are as follows:

<start> defaults to 0
<stop> defaults to the last + 1 index in the array. As python supports negative indexing (-ve indices “wrap around”, with -1 being the index after the largest index) this is often called the -1^th index, or that <stop> defaults to -1.
<step> defaults to 1

Therefore, the slice : selects all of the array, and the slice ::-1 selects all of the array, but reverses the ordering.

When dealing with N-dimensional arrays, indexing accepts a tuple. E.g., for a 2-dimensional array b=np.array([[10,11,15],[16,19,20],[33,35,36]]),

b[1,2] is equal to 20
b[2,1] is equal to 35

Similarly, slicing an N-dimensional array uses tuples of slices. E.g.,

>>> b
array([[10, 11, 15],
	   [16, 19, 20],
	   [33, 35, 36]])
>>> b[::-1,:]
array([[33, 35, 36],
	   [16, 19, 20],
	   [10, 11, 15]])
>>> b[1:2,::-1]
array([[20, 19, 16]])

Slices and indices can be mixed, so you can slice one dimension and select an index from another. E.g.,

>>> b[:1, 2]
array([15])
>>> b[::-1, 0]
array([33, 16, 10])
>>> b[0,::-1]
array([15, 11, 10])

There is a slice object in Python that can be used to programmatically create slices, its prototype is slice(start,stop,step), but only stop is required, and if stop=None the slice will continue until the end of the array dimension. Slice objects are almost interchangeable with the slice syntax. E.g.,

>>> s = slice(2)
>>> b[s,0]
array([10, 16])
>>> b[:2,0]
array([10, 16])

FITS File Format Information

Documentation for the Flexible Image Transport System (FITS) file format is hosted at NASA’s Goddard Space Flight centre, please refer to that as the authoritative source of information. What follows is a brief description of the format to aid understanding, see below for a schematic of a fits file.

A FITS file consists of one or more *header data units” (HDUs). An HDU contains header and (optionally) data information. The first HDU in a file is the “primary” HDU, and others are “extension” HDUs. The primary HDU always holds image data, extension HDUs can hold other types of data (not just images, but tables and anything else specified by the standard). An HDU always has a number which describes it’s order in the FITS file, and can optionally have a name. Naming an HDU is always a good idea as it helps users navigate the file. NOTE: The terms “extension”, “HDU”, and “backplane” are used fairly interchangeably to mean HDU

Within each HDU there is header-data and (optionally) binary-data. The header-data consists of keys and values stored as restricted ASCII strings of 80 characters in total. I.e., the whole key+value string must be 80 characters, they can be padded with spaces on the right. Practically, you can have as many header key-value entries as you have memory for. There are some reserved keys that define how the binary data of the HDU is to be interpreted. NOTE: Keys can only consist of uppercase latin letters, underscores, dashes, and numerals. The binary-data of an HDU is stored bin-endian, and intended to be read as a byte stream. The header-data describes how to read the binary-data, the most common data is image data and tabular data.

Fits image HDUs (and the primary HDU) define the image data via the following header keywords.

BITPIX: The magnitude is the number of bits in each pixel value. Negative values are floats, positive values are integers.
NAXIS: The number of axes the image data has, from 0->999 (inclusive).
NAXISn: The number of elements along axis “n” of the image.

Relating an axis to a coordinate system is done via more keywords that define a world coordinate system (WCS), that maps integer pixel indices to floating-point coordinates in, for example, time, sky position, spectral frequency, etc. The specifications for this are suggestions rather than rules, and non-conforming FITS files are not too hard to find. As details are what the spec is for, here is a high-level overview. Pixel indices are linearly transformed into “intermediate pixel coordinates”, which are rescaled to physical units as “intermediate world coordinates”, which are then projected/offset/have some (possibly non-linear) function applied to get them to “world coordinates”. The CTYPE keyword for an axis describes what kind of axis it is, i.e., sky-position, spectral frequency, time, etc.

Therefore, when using a FITS file it is important to specify which HDU (extension) to use, which axes of an image correspond to what physical coordinates, and sometimes what subset of the binary-data we want to operate upon.

Fits File Schematic

FITS FILE
|- Primary HDU
|  |- header data
|  |- binary data (optional)
|- Extension HDU (optional)
|  |- header data
|  |- binary data (optional)
|- Extension HDU (optional)
|  |- header data
|  |- binary data (optional)
.
.
.

FITS Data Order

FITS files use the FORTRAN convention of column-major ordering, whereas Python uses row-major ordering (sometimes called “C” ordering). For example, if we have an N-dimensional matrix, then we can specify a number in that matrix by its indices (e_1, e_2, e_3, …, e_N). FITS files store data so that the left most index changes the fastest, i.e., in memory the data is stored {a_11, a_21, a_31, …, a_M1, a_M2, …, a_ML}. However, Python stores its data where the right most index changes the fastest, i.e., data is stored in memory as {a_11, a_12, a_13, …, a_1L, a_2L, a_3L, …, a_NL}. Also, just to make things more difficult FITS (and Fortran) start indices at 1 whereas Python (and C) start indices at 0. The upshot of all of this is that if you have data in a FITS file, the axis numbers are related via the equation

N - f_i = p_i

where N is the number of dimensions, f_i is the FITS/FOTRAN axis number, p_i is the Python/C axis number.

The upshot is that even though both FITS and Python label the axes of an array from left-to-right, (the “leftmost” axis being 0 for python, 1 for FITS), the ordering of the data in memory means that when reading a FITS array in Python, the axes are reversed.

Example:

Let x be a 3 by 4 matrix, it has two axes. The FOTRAN convention is they are labelled 1 and 2, the C convention is that they are labelled 0 and 1. To make it obvious when we are talking about axes numbers in c or fortran I will use (f1, f2) for fortran, and (c0, c1) for c.

    / a b c d \
x = | e f g h |
    \ i j k l /

Assume x is stored as a 2 dimensional array.

In the FORTRAN convention, the matrix is stored in memory as {a e i b f j c g k d h l}, i.e., the COLUMNS vary the fastest

offset from start	0	1	2	3	4	5	6	7	8	9	10	11
value	a	e	i	b	f	j	c	g	k	d	h	l

Therefore, the memory offset of an element from the start of memory is m_i = (row-1) + number_of_rows * (column-1)

In FOTRAN, the left most index varies the fastest, and indices start from 1 so to extract a single number from x we index it via x[row, column] e.g., x[2,3] is an offset of (2-1)+3*(3-1) = 7, which selects ‘g’.

In the C convention, the matrix is stored in memory as {a b c d e f g h i j k l}, i.e., the ROWS vary the fastest

offset from start	0	1	2	3	4	5	6	7	8	9	10	11
value	a	b	c	d	e	f	g	h	i	j	k	l

Therefore, the memory offset of an element from the start of memory is m_i = number_of_columns * (row) + column

In C, the right most index varies the fastest, and indices start at 0 so to extract a single number from x we index it via x[row, column] E.g., x[1,2] is an offset of 4*1+2 = 6, which selects ‘g’ also.

Wait, these are the same (except the offset of 1)?! That is because we just looked at NATIVE data ordering. I.e., when FOTRAN and C have data they have written themselves.

What if we get C to read a FORTRAN written array?

The data in memory is stored as {a e i b f j c g k d h l}

If we index this the same way we did before, using x[row, column] we will have a problem. E.g., x[1,2] is an offset of 4*1+2 = 6, which selects ‘c’, not ‘g’!

This is happening because C assumes the axis that varies the fastest is the RIGHT-MOST axis. But this data is written so the fastest varing axis is the LEFT-MOST axis. So we should swap them around and use x[column, row]. E.g., For C; m_i = number_of_columns * (row) + column. Therefore, x[2,1] is an offset of 4*2+1 = 9, but that selects ‘d’, not ‘g’!?

This fails because we forgot to swap around the formula for the memory offset. The better way of writing the memory offset formula is

m_i = number_of_entries_in_fastest_varying_axis * (slowest_varying_axis_index) + fastest_varying_axis_index

And we know that for the data being written this way, the fastest varying axis has 3 entries not 4.

E.g., x[2,1] is an offset of 3*2+1 = 7, which that selects ‘g’, success!

What we have actually done is just make sure that we read the data the way it was written, in C the fastest varying axis is the RIGHT-MOST axis when indexing, so we have to reverse the indices AND the lengths of the axes. Therefore, data written in FORTRAN with axes (f1, f2, … fN) and axis lengths (K, L, …, M), should be read in C with axes ordered as (c0 = fN, c1 = fN-1, …, cN-2 = f2, cN-1 = f1) and lengths (M, …, L, K).

I.e., The fastest varying axis should always go at the correct position LEFT in FORTRAN and RIGHT in C.

Confusion happens because:

FORTRAN and C order axes from left-to-right
- FORTRAN starts at 1
- C starts at 0
The data order is different between FOTRAN and C
- FORTRAN writes data by varying the left-most axis the fastest
- C writes data by varying the right-most axis the fastest
FORTRAN and C index an N-dimensional native data order array the same way e.g., x[row, column]
- Axes numbers are based on the order in the written expression
- FORTRAN numbers these axes as row=1, column=2
- C numbers these axes as row=0, column=1
When reading data, it should always be read how it is ordered in memory for native data order this is as above, but for non native data order the axes order is reversed as it requires the axis speeds to be the same.
- C should read data as x[slowest, fastest]
- FORTRAN should read data as x[fastest, slowest]
- So x[row, column] written by FORTRAN is read as x[column, row] by C FORTRAN numbers them row=1, column=2; C numbers them column=0, row=1.
- In FOTRAN low axis numbers are fastest
- In C high axis numbers are fastest

NOTE: axis numbers are always from the left hand side.

FORTRAN	In memory	C axis
axis number	varying speed	number
N	slow	0
N-1	slower	1
…	…	…
2	fast	N-2
1	fastest	N-1

Code Snippets

Some useful code snippets are presented below.

Bash

Sudo Access Test

Enter the following commands at the command line:

ls
sudo ls

If, after entering your password, you see the same output for both commands, you have sudo access. Otherwise, you do not.

Python

Location of package source files

To find the location of the package’s files, run the following command:

python -c 'import site; print(site.getsitepackages())'

This will output the site packages directory for the python executable. The package’s files will be in the aopp_deconv_tool subdirectory.

Getting documentation from within python

Python’s command line, often called the “Read-Evaluate-Print-Loop (REPL)”, has a built-in help system.

To get the help information for a class, function, or object use the following code. Note, >>> denotes the python REPL (i.e., the command line you get when you type the python command), and $ denotes the shell command-line.

This example is for the built-in os module, but should work with any python object.

$ python
... # prints information about the python version etc. here
>>> import os
>>> help(os)
... # prints out the docstring of the 'os' module

Bash Scripts

Full Deconvolution Process

Below is an example bash script for performing every step of the deconvolution process on an observation and standard star file

#!/usr/bin/env bash

# Turn on "strict" mode
set -o errexit -o nounset -o pipefail

# Constants
THIS_SCRIPT=$0
USAGE="USAGE: whole_process.sh [-hr] <obs_fits:path> <std_fits:path> [slice:str] [spectral_axes:str] [celestial_axes:str]

Performs the entire deconvolution process from start to finish. Acts as a 
test-bed, an example bash script, and a way to use the tool without 
babysitting it.


# ARGUMENTS #

  obs_fits : path
	Path to the FITS file of the science observation to use, it will be 
	deconvolved at the end of this process.
  std_fits : path
	Path to the FITS file of the standard star observation to use, \`obs_fits\`
	will be deconvolved using (a model of) this as the PSF.


# OPTIONS #

  -h
	Display this help message
  -r
	Recalculate all products
  slice : str
	Python-style slice notation that will be applied to obs_fits and std_fits 
	data, often used to focus on specific spectral slice of data
  spectral_axes : str
	Axis number of spectral axis, enclosed in brackets e.g., '(0)'. Will be 
	automatically calculated if not present.
  celestial_axes : str
	Axis numbers of celestial axes, enclosed in brakcets e.g., '(1,2)'. Will be
	automatically calculated if not present.
"

# Functions
exit_with_msg() { echo "${@:2}"; exit $1; }
arg_error() { echo "${THIS_SCRIPT} ERROR: ${1}"; echo "${USAGE}"; exit 1; }

# START Parse Arguments
# NOTE: We just send the filenames, we rely on the defaults of the FITS Specifiers to handle extension and slice information for us
# Option defaults
RECALC=0

# let positional arguments and optional arguments be intermixed
# Therefore, must do this without useing "getopts"
N_REQUIRED_POS_ARGS=2
N_OPTIONAL_POS_ARGS=3
N_MAX_POS_ARGS=$((${N_REQUIRED_POS_ARGS}+${N_OPTIONAL_POS_ARGS}))
# echo "N_REQUIRED_POS_ARGS=${N_REQUIRED_POS_ARGS}"
# echo "N_OPTIONAL_POS_ARGS=${N_OPTIONAL_POS_ARGS}"
# echo "N_MAX_POS_ARGS=${N_MAX_POS_ARGS}"

declare -a POS_ARGS=()
ARGS=($@)
for ARG_IDX in ${!ARGS[@]}; do
	#echo "Processing argument at index ${ARG_IDX}"
	ARG=${ARGS[${ARG_IDX}]}
	#echo "    ${ARG}"
	#echo "#POS_ARGS[@]=${#POS_ARGS[@]}"
	case $ARG in
		-h)
			exit_with_msg 0 "${USAGE}"
			;;
		-r)
			RECALC=1
			;;
		*)
			if [[ ${#POS_ARGS[@]} -lt ${N_MAX_POS_ARGS} ]]; then
				POS_ARGS+=(${ARG})
			else
				arg_error "Maximum of ${N_MAX_POS_ARGS} positional arguments supported. Argument \"${ARG}\" is not an option or a positional."
			fi
			;;
	esac
done
if [[ ${#POS_ARGS[@]} -lt ${N_REQUIRED_POS_ARGS} ]]; then
	arg_error "Only ${#POS_ARGS[@]} positional arguments were specified, but ${N_REQUIRED_POS_ARGS} are required."
fi

#echo "POS_ARGS=${POS_ARGS[@]}"

# Get observation and standard star as 1st and 2nd argument to this script
FITS_OBS=${POS_ARGS[0]}
FITS_STD=${POS_ARGS[1]}
# Get slices, spectral axes, celestial axes as arguments 3,4,5
SLICE=${POS_ARGS[2]:-'[:]'}
SPECTRAL_AXES=${POS_ARGS[3]:-'(0)'}
CELESTIAL_AXES=${POS_ARGS[4]:-'(1,2)'}



# Output argument values for user information
echo "FITS_OBS=${FITS_OBS}"
echo "FITS_STD=${FITS_STD}"
echo "SLICE=${SLICE}"
echo "SPECTRAL_AXES=${SPECTRAL_AXES}"
echo "CELESTIAL_AXES=${CELESTIAL_AXES}"
echo "RECALC=${RECALC}"
# END Parse Arguments

# Set parameter constants

PSF_MODEL_STR="radial" # "radial" is the current default, it influences the name of the one of the output files
FITS_VIEWERS=("QFitsView" "ds9")

# Create output filenames for each step of the process that mirror the default output filenames

FITS_OBS_REBIN="${FITS_OBS%.fits}_rebin.fits"
FITS_OBS_REBIN_artefact="${FITS_OBS%.fits}_rebin_artefactmap.fits"
FITS_OBS_REBIN_artefact_BPMASK="${FITS_OBS%.fits}_rebin_artefactmap_bpmask.fits"
FITS_OBS_REBIN_INTERP="${FITS_OBS%.fits}_rebin_interp.fits"
FITS_OBS_REBIN_INTERP_DECONV="${FITS_OBS%.fits}_rebin_interp_deconv.fits"

FITS_STD_REBIN="${FITS_STD%.fits}_rebin.fits"
FITS_STD_REBIN_NORM="${FITS_STD%.fits}_rebin_normalised.fits"
FITS_STD_REBIN_NORM_MODEL="${FITS_STD%.fits}_rebin_normalised_modelled_${PSF_MODEL_STR}.fits"


ALL_FITS_FILES=(
	${FITS_OBS_REBIN} 
	${FITS_STD_REBIN}
	${FITS_OBS_REBIN_artefact}
	${FITS_OBS_REBIN_artefact_BPMASK}
	${FITS_OBS_REBIN_INTERP}
	${FITS_STD_REBIN_NORM}
	${FITS_STD_REBIN_NORM_MODEL}
	${FITS_OBS_REBIN_INTERP_DECONV}
)

# Perform each stage in turn

echo "Performing spectral rebinning"
if [[ ${RECALC} == 1 || ! -f ${FITS_OBS_REBIN} ]]; then
	python -m aopp_deconv_tool.spectral_rebin "${FITS_OBS}${SLICE}${SPECTRAL_AXES}"
fi
if [[ ${RECALC} == 1 || ! -f ${FITS_STD_REBIN} ]]; then
	python -m aopp_deconv_tool.spectral_rebin "${FITS_STD}${SLICE}${SPECTRAL_AXES}"
fi

echo "Performing artefact detection"
if [[ ${RECALC} == 1 || ! -f ${FITS_OBS_REBIN_artefact} ]]; then
	python -m aopp_deconv_tool.artefact_detection "${FITS_OBS_REBIN}${SLICE}${CELESTIAL_AXES}"
fi

echo "Creating bad pixel mask"
if [[ ${RECALC} == 1 || ! -f ${FITS_OBS_REBIN_artefact_BPMASK} ]]; then
	python -m aopp_deconv_tool.create_bad_pixel_mask "${FITS_OBS_REBIN_artefact}${SLICE}${CELESTIAL_AXES}"
fi

echo "Interpolating at bad pixel mask"
if [[ ${RECALC} == 1 || ! -f ${FITS_OBS_REBIN_INTERP} ]]; then
	python -m aopp_deconv_tool.interpolate "${FITS_OBS_REBIN}${SLICE}${CELESTIAL_AXES}" "${FITS_OBS_REBIN_artefact_BPMASK}${SLICE}${CELESTIAL_AXES}"
fi

echo "Normalising PSF"
if [[ ${RECALC} == 1 || ! -f ${FITS_STD_REBIN_NORM} ]]; then
	python -m aopp_deconv_tool.psf_normalise "${FITS_STD_REBIN}${SLICE}${CELESTIAL_AXES}"
fi

echo "Modelling PSF"
if [[ ${RECALC} == 1 || ! -f ${FITS_STD_REBIN_NORM_MODEL} ]]; then
	python -m aopp_deconv_tool.fit_psf_model "${FITS_STD_REBIN_NORM}${SLICE}${CELESTIAL_AXES}" --model "${PSF_MODEL_STR}"
fi

echo "Performing deconvolution"
if [[ ${RECALC} == 1 || ! -f ${FITS_OBS_REBIN_INTERP_DECONV} ]]; then
	python -m aopp_deconv_tool.deconvolve "${FITS_OBS_REBIN_INTERP}${SLICE}${CELESTIAL_AXES}" "${FITS_STD_REBIN_NORM_MODEL}${SLICE}${CELESTIAL_AXES}"
fi

echo "Deconvolved file is ${FITS_OBS_REBIN_INTERP_DECONV}"

# Open all products in the first viewer available
for FITS_VIEWER in ${FITS_VIEWERS[@]}; do
	if command -v ${FITS_VIEWER} &> /dev/null; then
		${FITS_VIEWER} ${ALL_FITS_FILES[@]} &
		break
	fi
done

Linux Python Installation

Below is an example bash script for building python from source and configuring a virtual environment. Use it via copying the code into a file (recommended name install_python.sh). If Python’s dependencies are not already installed, you will need sudo access so the script can install them.

Make the script executable : chmod u+x install_python.sh
Get help on the scripts options with: ./install_python.sh -h
Run the script with : ./install_python.sh

#!/usr/bin/env bash

# Turn on "strict" mode
set -o errexit -o nounset -o pipefail

## Remember values of environment variables as we enter the script
OLD_IFS=$IFS 
INITIAL_PWD=${PWD}

## Define Constants
TRUE=0
FALSE=1

## Define Variables that don't depend on arguments
# Size of subarea to display output of long-running commands, set to 0 to disable.
TERMINAL_SUBAREA_SIZE=20
# Flag to set terminal back to previous state
TERM_DIRTY_FLAG=${FALSE}
# For keeping track of when we should exit.
EXIT_FLAG=${FALSE}


############################################################################################
##############                    PROCESS ARGUMENTS                         ################
############################################################################################

# Set default parameters
PYTHON_VERSION=(3 12 2)
PYTHON_INSTALL_DIRECTORY="${HOME:?}/.local/.python"
VENV_PREFIX=".venv_"
VENV_DIR="${PWD}"
LOG_FILE="${PYTHON_INSTALL_DIRECTORY}/install_<python_version>.log"
PYTHON_SOURCE_DIR=""

# Flags for default parameters
LOG_FILE_SET=${FALSE}
SHOW_HELP=${FALSE}
SHOW_CV_HELP=${FALSE}

# Get the usage string with the default values of everything
usage(){
        local param_value_type_string=${1:-Current Value}
        {
                echo "install_python.sh [-v INT.INT.INT] [-i PATH] [-p STR] [-d PATH] [-l PATH] [-s PATH] [-h] [-H]"
                echo ""
                print_param_info "${param_value_type_string}"
        } | text_frame_around '-' '|' 'USAGE'
}

text_n_str(){
        local n=$1
        local str=${2}

        for ((i=0; i<n; i++)); do echo -n "${str}"; done
}

text_frame_around() {
        local hchar=${1:--}
        local vchar=${2:-|}
        local title=${3:-}

        local ifs_old="${IFS}"

        if [ -n "${title}" ]; then
                title=" ${title} "
        fi

        local n_title=$(echo "${title}" | wc -L)
        local text_str="$(cat -)"
        local lmax=$(wc -L <<<"${text_str}")

        if [ $((lmax+4)) -lt ${n_title} ]; then
                lmax=$((n_title+2))
        else
                lmax=$((lmax + 4))
        fi

        local remainder=${lmax}

        # First line
        echo -n "${hchar}"
        echo -n "${title}"
        text_n_str $((lmax - 1 - n_title)) ${hchar}
        echo ""

        # Empty line
        echo -n ${vchar}
        text_n_str $((lmax -2)) ' '
        echo ${vchar}

        # Content
        while read; do
                remainder=$(wc -L <<<"$REPLY")
                remainder=$((lmax -2 - remainder))
                echo -n "${vchar} "
                echo -n "$REPLY"
                text_n_str $((remainder -1)) ' '
                echo "${vchar}"

        done <<<"$text_str"

        # Empty line
        echo -n ${vchar}
        text_n_str $((lmax -2)) ' '
        echo ${vchar}

        # Final line
        text_n_str ${lmax} ${hchar}
        echo ""

}

print_param_info() {
        local param_value_type_string=${1:-Current Value}

        echo "    -v : PYTHON_VERSION <INT.INT.INT>"
        echo "         Python version to install."
        echo "         ${param_value_type_string} = ${PYTHON_VERSION[0]}.${PYTHON_VERSION[1]}.${PYTHON_VERSION[2]}"
        echo ""
        echo "    -i : PYTHON_INSTALL_DIRECTORY <PATH>"
        echo "         Path to install python to."
        echo "         ${param_value_type_string} = '${PYTHON_INSTALL_DIRECTORY}'"
        echo ""
        echo "    -p : VENV_PREFIX <STR>"
        echo "         Prefix for virtual environment (will have python version added as a suffix)."
        echo "         ${param_value_type_string} = '${VENV_PREFIX}'"
        echo ""
        echo "    -d : VENV_DIR <PATH>"
        echo "         Directory to create virtual envronment in, if empty will not create a virtual environment."
        echo "         ${param_value_type_string} = '${VENV_DIR}'"
        echo ""
        echo "    -l : LOG_FILE <PATH>"
        echo "         File to copy output to. If empty will only output to stdout (the terminal)."
        echo "         ${param_value_type_string} = '${LOG_FILE}'"
        echo ""
        echo "    -s : PYTHON_SOURCE_DIR <PATH>"
        echo "         Directory to download source to, will be a temp directory if not set. "
        echo "         ${param_value_type_string} = '${PYTHON_SOURCE_DIR}'"
        echo ""
        echo "    -h : Display this help message with default parameter values"
        echo ""
        echo "    -H : Display this help message with passed parameter values"
        echo ""
}


USAGE=$(usage "Default")

# Parse input arguments
while getopts "v:i:p:d:l:s:hH" OPT; do
        case $OPT in
                v)
                        IFS="."
                        PYTHON_VERSION=(${OPTARG})
                        IFS=$OLD_IFS
                        ;;
                i)
                        PYTHON_INSTALL_DIRECTORY=${OPTARG}
                        ;;
                p)
                        VENV_PREFIX=${OPTARG}
                        ;;
                d)
                        VENV_DIR=${OPTARG}
                        ;;
                l)
                        LOG_FILE=${OPTARG}
                        LOG_FILE_SET=${TRUE}
                        ;;
                s)
                        PYTHON_SOURCE_DIR=${OPTARG}
                        ;;
                H)
                        SHOW_CV_HELP=${TRUE}
                        ;;
                *)
                        SHOW_HELP=${TRUE}
                        ;;
        esac
done

## Perform argument processing

PYTHON_VERSION_STR="${PYTHON_VERSION[0]}.${PYTHON_VERSION[1]}.${PYTHON_VERSION[2]}"

# If the log file was not set on the command-line, use a default location
if [ ${LOG_FILE_SET} -eq ${FALSE} ]; then
        LOG_FILE="${PYTHON_INSTALL_DIRECTORY}/install_${PYTHON_VERSION_STR}.log"
fi

# If the python source directory is not specified, use a temporary directory
if [ -z "${PYTHON_SOURCE_DIR}" ]; then
        TEMP_WORKSPACE=$(mktemp -d -t py_build_src.XXXXXXXX)
        PYTHON_SOURCE_DIR="${TEMP_WORKSPACE}"
else
        TEMP_WORKSPACE=""
fi

# After any processing, show help message if required.
if [ ${SHOW_HELP} -eq ${TRUE} ]; then
        echo "${USAGE}"
        EXIT_FLAG=${TRUE}
fi
if [ ${SHOW_CV_HELP} -eq ${TRUE} ]; then
        echo "$(usage)"
        EXIT_FLAG=${TRUE}
fi
if [ ${EXIT_FLAG} -eq ${TRUE} ]; then
        exit 0
fi

# Print parameters to user so they know what's going on
echo "Parameters:"
print_param_info


############################################################################################
##############                     DEFINE FUNCTIONS                         ################
############################################################################################


install_pkg_if_not_present(){

        # Turn on "strict" mode
        set -o errexit -o nounset -o pipefail
        REQUIRES_INSTALL=()

        for PKG in "$@"; do
                # We want the command to fail when a package is not installed, therefore unset errexit
                set +o errexit 
                        DPKG_RCRD=$(dpkg-query -l ${PKG} 2> /dev/null | grep "^.i.[[:space:]]${PKG}\(:\|[[:space:]]\)")
                        INSTALLED=$?
                set -o errexit

                if [ ${INSTALLED} -eq 0 ]; then
                        echo "  ${PKG} is installed"
                else
                        echo "  ${PKG} is NOT installed"
                        REQUIRES_INSTALL[${#REQUIRES_INSTALL[@]}]=${PKG}
                fi

        done


        if [ ${#REQUIRES_INSTALL[@]} -ne 0 ]; then


                UNFOUND_PKGS=()
                for PKG in ${REQUIRES_INSTALL[@]}; do
                        # We want the command to fail when a package is not installed, therefore unset errexit
                        set +o errexit 
                                apt-cache showpkg ${PKG} | grep -E "^Package: ${PKG}"
                                PKG_FOUND=$?
                        set -o errexit

                        if [ $PKG_FOUND -ne 0 ]; then
                                echo "Could not find package '${PKG}' using 'apt-cache showpkg'"
                                UNFOUND_PKGS[${#UNFOUND_PKGS[@]}]=${PKG}
                        fi
                done

                if [ ${#UNFOUND_PKGS[@]} -ne 0 ]; then 
                        echo "ERROR: Cannot install. Could not find the following packages in apt: ${UNFOUND_PKGS[@]}"
                        return 1
                fi

                echo "Installing packages: ${REQUIRES_INSTALL[@]}"
                sudo apt-get install -y ${REQUIRES_INSTALL[@]}
        else
                echo "All required packages are installed"
        fi
}

set_term_scroll_region(){
        TERM_DIRTY_FLAG=${TRUE}

        local n_lines=$(tput lines)
        local n_scroll=${1:-${TERMINAL_SUBAREA_SIZE:-$((lines/5))}}
        local n_after=2
        local scroll_to=$((n_lines - 1 - n_after))
        local scroll_from=$((scroll_to - n_scroll))

        for ((i=0; i<(n_scroll+1+n_after); i++)); do echo ""; done

        tput cup $((scroll_from -1)) 0
        echo "-------------------------------------------------------------------------"
        tput cup $((scroll_to +1)) 0
        echo "-------------------------------------------------------------------------"

        tput csr $scroll_from $scroll_to

        tput cup $scroll_from 0

}

unset_term_scroll_region(){
        n_lines=$(tput lines)
        #tput smcup
        tput csr 0 $((n_lines -1))
        #tput rmcup
        tput cup $n_lines
        TERM_DIRTY_FLAG=${FALSE}
}

term_subarea(){
        set_term_scroll_region
        cat -
        unset_term_scroll_region
}

term_remove_ctrl_chars(){
        sed -u 's/\x1B[@A-Z\\\]^_]\|\x1B\[[0-9:;<=>?]*[-!"#$%&'"'"'()*+,.\/]*[][\\@A-Z^_`a-z{|}~]//g'
}


install_python(){

        echo "Checking python dependencies and installing if required..."
        install_pkg_if_not_present ${PYTHON_DEPENDENCIES[@]} | term_subarea

        if [ -f "${PY_SRC_FILE}" ]; then
                echo "Source for python version ${PYTHON_VERSION_STR} already exists at ${PY_SRC_FILE}"
        else
                echo "Downloading python source code to '${PY_SRC_FILE}'..."
                mkdir -p ${PYTHON_SOURCE_DIR}
                curl ${PYTHON_VERSION_SOURCE_URL} --output ${PY_SRC_FILE}
        fi

        if [ -f "${PY_SRC_DIR}/configure" ]; then
                echo "Python source already extracted to ${PY_SRC_DIR}"
        else
                echo "Extracting source file..."
                mkdir -p ${PY_SRC_DIR}
                tar -xvzf ${PY_SRC_FILE} -C ${PYTHON_SOURCE_DIR} | term_subarea
        fi

        cd ${PY_SRC_DIR}
        echo "Configuring python installation..."
        ./configure                                  \
                --prefix=${PYTHON_VERSION_INSTALL_DIR:?} \
                --enable-optimizations                   \
                --with-lto                               \
                --enable-ipv6                            \
                --with-computed-gotos                    \
                --with-system-ffi                        \
                --disable-test-modules                   \
                --with-ensurepip=install                 \
                | term_subarea



        echo "Running makefile..."
        if command -v nproc &> /dev/null; then
                # Spread across all processors if possible
                make -j $(($(nproc)-1)) | term_subarea
        else
                make | term_subarea
        fi

        echo "Created ${PYTHON_VERSION_INSTALL_DIR}"
        mkdir -p ${PYTHON_VERSION_INSTALL_DIR}

        echo "Performing installation..."
        if [ -e "${PYTHON_VERSION_INSTALL_DIR}/bin/python3" ]; then
                echo "NOTE: '${PYTHON_VERSION_INSTALL_DIR}/bin/python3' already exists, using 'altinstall'"
                echo "----:  to ensure we do not overwrite it as it may be the system Python"
                make altinstall                              \
                        | term_subarea
        else
                make install                                 \
                        | term_subarea
        fi
        cd ${INITIAL_PWD}
}

create_virtual_environment() {

        if [ -z "${VENV_DIR}" ]; then
                echo "No virtual environment created."
                return
        fi

        echo "Creating virtual environment..."
        ${PYTHON_VERSION_INSTALL_DIR}/bin/python3 -m venv ${VENV_PATH}

        echo "Virtual environment created at ${VENV_PATH}"


        # Output information to user
        echo ""
        echo "Activate the virtual environment with the following command:"
        echo "    source ${VENV_PATH}/bin/activate"
} 


############################################################################################
##############                       START SCRIPT                           ################
############################################################################################

# Define the dependencies that python requires for installation
PYTHON_DEPENDENCIES=(   \
        curl                \
        gcc                 \
        make                \
        wget                \
        xz-utils            \
        tar                 \
        llvm                \
        libbz2-dev          \
        libev-dev           \
        libffi-dev          \
        libgdbm-dev         \
        liblzma-dev         \
        libncurses-dev      \
        libreadline-dev     \
        libsqlite3-dev      \
        libssl-dev          \
        tk-dev              \
        zlib1g-dev          \
)

if commmand -v apt &> /dev/null; then
        {
                echo "Could not install Python for this machine.                       "
                echo "                                                                 "
                echo "This script is created for Linux distributions that use the 'apt'"
                echo "package distribution program, i.e.. Debian based distributions   "
                echo "such as Ubuntu.                                                  "
                echo "                                                                 "
                echo "Distributions with other package managers should use a different "
                echo "method of installing alternate Python versions.                  "
                echo "                                                                 "
                echo "For Example:                                                     "
                echo "                                                                 "
                echo "  * The official site (https://www.python.org/downloads/) has    "
                echo "    installers for Windows and MacOs                             "
                echo "                                                                 "
                echo "  * This article (https://realpython.com/installing-python/) has "
                echo "    instructions for Windows, MacOs. and Linux distributions     "
                echo "                                                                 "
                echo "  * This site (https://www.build-python-from-source.com/) has    "
                echo "    installation scripts for a variety of Linux distributions.   "
        } | text_frame_around '#' '#' 'WARNING'
        exit ${FALSE}
fi


# Get a temporary directory and make sure it's cleaned up when the script exits
cleanup(){
        echo "Cleaning up on exit..."

        echo ""
        unset_term_scroll_region

        if [ -n "${TEMP_WORKSPACE}" ] && [ -e "${TEMP_WORKSPACE}" ]; then
                echo "Removing ${TEMP_WORKSPACE} ..."
                rm -rf ${TEMP_WORKSPACE:?}
        fi
}
trap cleanup EXIT SIGTERM

# If there is an error, make sure we print the usage string with default parameter values
error_message(){
        echo "${USAGE}"
}
trap error_message ERR


# Define variables

PYTHON_VERSION_INSTALL_DIR="${PYTHON_INSTALL_DIRECTORY}/${PYTHON_VERSION_STR}"
VENV_PATH="${VENV_DIR}/${VENV_PREFIX}${PYTHON_VERSION_STR}"
PYTHON_VERSION_SOURCE_URL="https://www.python.org/ftp/python/${PYTHON_VERSION_STR}/Python-${PYTHON_VERSION_STR}.tgz"

PY_SRC_DIR="${PYTHON_SOURCE_DIR}/Python-${PYTHON_VERSION_STR}"
PY_SRC_FILE="${PY_SRC_DIR}.tgz"

# Create directories if required
mkdir -p "${PYTHON_INSTALL_DIR}"

# Perform actions
install_python | tee >(term_remove_ctrl_chars > ${LOG_FILE:-/dev/null})
create_virtual_environment | tee >(term_remove_ctrl_chars > ${LOG_FILE:-/dev/null})

Table of Contents