Table of Contents

Development

Development Environment Setup

This section details how to set up a development environment for this project.

placeholder definitions

Placeholders:

`<REPO_DIR>`
	The top-level directory of this repository. I.e., the directory this file is in.

`<X.Y.Z>`
	The version of python used in development, currently this is `3.12.2`.

`<VENV_DIR>`
	The directory of the python virtual environment for the project. Default name is `.venv_<PYTHON_VERSION>`.

setting up the virtual environment

default installation as a single command

Use the following command to install the desired version of python and do the steps above.

PYTHON_VERSION=(3 12 2) && \
PYTHON_INSTALL_DIRECTORY="${HOME:?}/python/python_versions" && \
REPO_DIR="${HOME:?}/repos/aopp_obs_toolchain" && \
VENV_PREFIX=".venv" && \
echo "" && \
echo "PYTHON_MAJOR_VERSION=${PYTHON_VERSION[0]:?ERROR: PYTHON_VERSION must have 3 entries (e.g., '(3 12 2)')}" && \
echo "PYTHON_MINOR_VERSION=${PYTHON_VERSION[1]:?ERROR: PYTHON_VERSION must have 3 entries (e.g., '(3 12 2)')}" && \
echo "PYTHON_BUILD_VERSION=${PYTHON_VERSION[2]:?ERROR: PYTHON_VERSION must have 3 entries (e.g., '(3 12 2)')}" && \
echo "PYTHON_INSTALL_DIRECTORY=${PYTHON_INSTALL_DIRECTORY:?ERROR: PYTHON_INSTALL_DIRECTORY must be set}" && \
echo "REPO_DIR=${REPO_DIR:?ERROR: REPO_DIR must be set}" && \
echo "VENV_PREFIX=${VENV_PREFIX:?ERROR: VENV_PREFIX must be set}" && \
echo "" && \
PYTHON_VERSION_STR="${PYTHON_VERSION[0]}.${PYTHON_VERSION[1]}.${PYTHON_VERSION[2]}" && \
echo "${PYTHON_VERSION_STR}" && \
VENV_DIR="${REPO_DIR:?}/${VENV_PREFIX:?}_${PYTHON_VERSION_STR:?}" && \
DOWNLOAD_URL="https://www.python.org/ftp/python/${PYTHON_VERSION_STR:?}/Python-${PYTHON_VERSION_STR:?}.tgz" && \
PYTHON_VERSION_INSTALL_DIR="${PYTHON_INSTALL_DIRECTORY:?}/python${PYTHON_VERSION_STR:?}" && \
echo "PYTHON_VERSION_INSTALL_DIR=${PYTHON_VERSION_INSTALL_DIR:?}" && \
mkdir -p ${PYTHON_VERSION_INSTALL_DIR:?} && \
PYTHON_VERSION_SOURCE_DIR="${PYTHON_VERSION_INSTALL_DIR:?}/Python-${PYTHON_VERSION_STR:?}" && \
echo "PYTHON_VERSION_SOURCE_DIR=${PYTHON_VERSION_SOURCE_DIR:?}" && \
PYTHON_VERSION_DOWNLOAD_FILE="${PYTHON_VERSION_SOURCE_DIR:?}.tgz" && \
echo "PYTHON_VERSION_DOWNLOAD_FILE=${PYTHON_VERSION_DOWNLOAD_FILE:?}" && \
echo "Downloading python source from ${PYTHON_VERSION_DOWNLOAD_FILE:?}..." && \
curl ${DOWNLOAD_URL:?} --output ${PYTHON_VERSION_DOWNLOAD_FILE:?} && \
echo "Python source downloaded." && \
echo "Installing dependencies..." && \
sudo apt-get install -y \
	curl \
	gcc \
	libbz2-dev \
	libev-dev \
	libffi-dev \
	libgdbm-dev \
	liblzma-dev \
	libncurses-dev \
	libreadline-dev \
	libsqlite3-dev \
	libssl-dev \
	make \
	tk-dev \
	wget \
	zlib1g-dev && \
echo "Dependencies installed." && \
echo "Extracting source file ${PYTHON_VERSION_DOWNLOAD_FILE:?}" && \
tar -xvzf ${PYTHON_VERSION_DOWNLOAD_FILE:?} -C ${PYTHON_VERSION_INSTALL_DIR:?} && \
echo "source file extacted." && \
echo "Moving into source-code directory. ${PYTHON_VERSION_SOURCE_DIR:?}..." && \
cd ${PYTHON_VERSION_SOURCE_DIR:?} && \
echo "Configuring python installation..." && \
./configure \
	--prefix=${PYTHON_VERSION_INSTALL_DIR:?} \
	--enable-optimizations \
	--enable-ipv6 \
	 && \
echo "Configuration done." && \
echo "Running makefile..." && \
make && \
echo "Makefile complete." && \
echo "Performing installation..." && \
make install && \
echo "Installation complete" && \
echo "Making virtual environment at ${VENV_DIR:?}" && \
${PYTHON_VERSION_INSTALL_DIR:?}/bin/python3 -m venv ${VENV_DIR:?} && \
echo "Virtual environment created" && \
PYTHON_MAJOR_MINOR_STR="${PYTHON_VERSION[0]:?}.${PYTHON_VERSION[1]:?}" && \
echo "PYTHON_MAJOR_MINOR_STR=${PYTHON_MAJOR_MINOR_STR:?}" && \
REPO_SOURCE_DIR=$(readlink -f "${REPO_DIR:?}/src") && \
echo "REPO_SOURCE_DIR=${REPO_SOURCE_DIR:?}" && \
echo "Creating '.pth' file for virtual environment..." && \
echo "${REPO_SOURCE_DIR:?}" > ${VENV_DIR:?}/lib/python${PYTHON_MAJOR_MINOR_STR:?}/site-packages/aopp_obs_toolchain.pth && \
echo "'.pth' file created" && \
echo "Adding aliases to avoid PYTHONPATH..." && \
echo -e "alias python=\"python -E\"\nalias python3=\"python3 -E\"\nalias python${PYTHON_MAJOR_MINOR_STR:?}=\"python${PYTHON_MAJOR_MINOR_STR:?} -E\"" >> ${VENV_DIR:?}/bin/activate && \
echo "Aliases added." && \
echo "Activating virtual environment..." && \
source ${VENV_DIR:?}/bin/activate && \
echo "Virtual environment activated" && \
echo "Installing required packages" && \
pip install -r ${REPO_DIR:?}/requirements.txt && \
echo "Required packages installed"

VSCode Setup

If using WSL (Windows Subsystem for Linux)

Getting Plots To Display Correctly

If Using Linux (Including WSL)

Getting the virtual environment to activate automatically

Often the way VSCode activates virtual environments is not the “normal” way, so things like *.pth files are not treated correctly, to get around this problem don’t let VSCode activate virtual environments for itself, and add a check to your ~/.bashrc file to look for a virtual environment.

	"terminal.integrated.env.linux": {
		"VSCODE_WORKSPACE_DIR" : "${workspaceFolder}"
	}

This adds the environment variable “VSCODE_WORKSPACE_DIR” to every terminal that VSCode opens, and sets it to the current workspace (i.e., top level) folder.

Running Tests

The tests are in the directory <REPO_DIR>/tests, there is a package <REPO_DIR>/scientest which is a testing tool. The module <REPO_DIR>/scientest/run.py will search for tests and run them one by one. It tries to ensure that tests do not have side-effects.

Folders are searched if:

* They **do not**  begin with double underscores (`__`).

Files are searched if:

* They have `test` in their name.

* They end with `.py`.

* They **do not** begin with double underscores (`__`).

Steps to run tests

Test Output

Building the Package

Run the command python -m build from the <REPO_DIR> directory. The <REPO_DIR>/dist and <REPO_DIR>/aopp_deconv_tool.egg-info folders should be created. These contain the built package files.

NOTE: When rebuilding, you may want to use rm ./dist/*; python -m build instead, as otherwise the previous build information will stick around.

Uploading the Package to Pypi

From the <REPO_DIR> directory, run ONE of the following commands:

Use __token__ for <USERNAME>, and an API Token value for <PASSWORD> (including the pypi- prefix). See this guide for uploading to the package index for more information.

If storing the token in a file, use python3 -m twine upload -u __token__ -p $(cat ~/.secrets/pypi_token) dist/*.

Verify the package uploaded correctly by going to https://test.pypi.org/project/aopp-deconv-tool/ or https://pypi.org/project/aopp-deconv-tool/.

Storing The PyPi Token

You could try remembering the PyPi token (well done if you can), however that’s not often possible. You don’t want to hard-code the token or check it into git as other people will have different ones and then the token would be visible to anyone with access to the repository (which can be public). There are a few different ways to solve the problem, I’ll go through some of them

Option 1: Use a password manager of some sort. This isn’t too bad, you put all API tokens in a password manager program and just need to remember the master password. You can get versons that sync across devices. The annoying thing is that you often have to go in and decrypt and copy the token manually as you really don’t want to let the master password be stored somewhere.

Option 2: Set an environment variable. This can work quite well, you set an environment variable (in your ~/.basrhc file for example, not somewhere that is checked into git), then use that environment variable wherever you need it. It’s nice and convenient, but you do have the token sitting in plain text somewhere and always loaded up in memory so if you ever need to send your environment variables to IT for debugging, your token gets sent as well unless you remember to remove it. Also a child process inherits it’s parents environment variables, so you technically have to be sure that nothing else is sending environment variables to random places. However, it’s easy to use as you just need to add ${PYPI_TOKEN} in place of the token whenever you need to use it.

Option 3: Store in a (preferably encrypted) file, not in the repository. This is sort-of half-way between (1) and (2). You store the password in a file (e.g., ~/.secrets/pypi_token) and make sure that it is:

Then use $(cat ~/.secrets/pypi_token) inplace of the token whenever you need to use it.

Personally, I find option 3 to be the best. Even without encryption, someone would either need root access or physical access to the drive to get the token if the directory and file permissions are set up correctly.

Test the package uploaded correctly

Create a NEW virtual environment to test the package in. If you use the one in this repository it will just use the defaults we have set up for development.

If using the TEST python package index, run the following command:

If using the REAL python package index, run the following command:

run the examples using the newly installed package

In whatever test directory you want, run the examples with:

These should all complete and write files to their outputs just as if you ran them in the development environment. You might want to delete the contents of the output directories before running so you can see the new files being created.

Documentaion

DOXYGEN Documentation

To generate the DOXYGEN documentation:

Pydoctor Documentation

The pydoctor package is part of the build requirements for this package, so you should already have it available in your python development environment for this package.

To build the documentation, run the following command in the <REPO> directory:

python -m pydoctor ./src/aopp_deconv_tool --project-name aopp_obs_toolchain::aopp_deconv_tool --project-base-dir ./src/aopp_deconv_tool --make-html --docformat google --html-output ./www/documentation/pydoc --sidebar-expand-depth 5 --theme readthedocs

View the documentation by opening <REPO>/www/documentation/pydoc/index.html in your browser.

Command argument explanation:

./src/aopp_deconv_tool
The folder of the package to create documentation for.
--project-name aopp_obs_toolchain::aopp_deconv_tool
Defines the name of the project, aopp_deconv_tool is part of the aopp_obs_toolchain package.
--project-base-dir ./src/aopp_deconv_tool
All paths in the documentation are relative to this path.
--make-html
Output HTML so we can display the documentation as a web page.
--docformat google
Documentation formatting follows google’s guidelines (mostly)
--html-output ./www/documentation/pydoc
Folder to store HTML output in, in this case somewhere we can point the user towards on the github pages site.
--sidebar-expand-depth 5
Depth to which the sidebar will expand when navigating nested packages etc.
--theme readthedocs
Formatting and styling theme to use, “readthedocs” is pretty good.