Welcome to the Algorithm Toolkit (ATK) documentation!¶

This project provides an easy way for researchers and developers to develop and share algorithms related to geospatial data and imagery.

The source code for this project lives on GitHub.

Contents¶

About the Algorithm Toolkit¶

Why does this exist?¶

Algorithms on this site can benefit people and industries. From helping investors predict markets more accurately to making farmers more productive, image processing algorithms are revolutionizing the way we live and work. For too long these have been locked away in academia or in private companies. We thought it was time to bring them to the world.

What are algorithms?¶

Images are data. For each pixel in an image you may have hundreds or even thousands of data values captured by sophisticated sensor equipment, or attached by software during image processing. Algorithms are basically mathematical equations applied to the data contained in images in order to reveal aspects of the image that may not normally be visible.

What is the ATK?¶

Researchers all over the world have written software programs that apply algorithms to images. Up to now, those programs have been written in lots of different ways using lots of different software tools, and haven’t been able to talk to other programs written by other researchers. Our ATK creates a way for anyone who develops one of these software programs to make it available in a standard way.

Doesn’t sound like such a big deal.¶

Typically, many algorithms are used in sequence to process an image. For example, if you want to show vegetation health from pictures taken from a satellite, you might:

Download the pictures from the satellite’s feed
Stitch together the pictures into a single orthomosaic image
Perform a calculation called NDVI to show amounts of chlorophyll in each pixel of that image
Apply a color scale to the resulting pixels (which otherwise would be grayscale); maybe green to red
Export the image to the web

Each of these steps might be done by a single algorithm, but how does each step know what to do with the stuff that came before it? And how does it know what to provide the next step?

The ATK creates a way to define an algorithm “chain” with standard inputs and outputs. That way, when you create an algorithm you know what you’ll be getting in and can tell other developers what you’re providing out.

Installation¶

Super Quick Start¶

Python projects should be installed in virtual environments in order to keep versions of various packages in sync with the project code. However, if you want to get up and running immediately you can do the following in a Terminal window (assumes Python and pip are installed):

pip install algorithm_toolkit
alg cp myproject
cd myproject
alg run

Point your browser to http://localhost:5000/. You should see the development environment welcome page.

See the next section for a more detailed install process.

Slightly More Involved But Still Pretty Quick Start¶

A word about virtual environments¶

As noted above, it’s a best practice to start new Python projects in virtual environments. As you work on code, you rely more and more on external libraries. As time passes, changes to those libraries will eventually break your code unless you are constantly updating it. So when you install a new version of a library for another project, suddenly you find that your earlier project no longer works.

Virtual environments solve this problem. Each environment contains a discreet set of the libraries used by your code, at the versions you determine.

Thankfully, creating virtual environments in Python is easy.

Python 3¶

Linux and Mac:

python3 -m venv myenvironment
source myenvironment/bin/activate

Note

On some Linux systems, python3-venv is not installed by default. If you get an error with the command above, try:

# Debian/Ubuntu
sudo apt install python3-venv

# RedHat/CentOS
sudo yum install python3-venv

On Windows:

py -3 -m venv myenvironment
myenvironment\Scripts\activate.bat

Python 2¶

Python 2 does not come with a built in virtual environment creator. We recommend using virtualenvwrapper. Their install docs are excellent.

Once you’ve installed and configured virtualenvwrapper, create your virtual environment:

mkvirtualenv myenvironment

Setting up your ATK project¶

pip install algorithm_toolkit
alg cp myproject
cd myproject
alg run

Point your browser to http://localhost:5000/. You should see the development environment welcome page.

What just happened?¶

Let’s walk through this line by line.

pip install algorithm_toolkit

The ATK lives on PyPi, so this line downloads and installs the ATK in your virtual environment. Several dependencies will be installed as well.

alg cp myproject

This line uses the ATK’s Command Line Interface (CLI) called alg. The cp command stands for “create project”. “myproject” is the name of your project, which will also be the name of the folder created for the project (feel free to use a more original name).

cd myproject

Puts you in the project folder.

alg run

This command also uses the CLI. The run command starts a development web server. As we discuss elsewhere in the docs (see TODO: create page), setting up your algorithms and processing chains is accomplished using a web-based interface.

Installing the example project¶

The ATK comes with an example project to help you understand how it works.

Prerequisites¶

The example project has some additional dependencies, including NumPy, in order to work. If you’re on a Linux machine, you can install the example project and it will handle this dependency for you.

However, if you’re on a Mac or Windows machine, installing NumPy is more complicated.

Anaconda¶

For these operating systems, we highly recommend using Anaconda or it’s smaller cousin Miniconda. The only difference between these two is that Anaconda installs over 150 packages (including SciPy and NumPy) out of the box whereas with Miniconda you need to install everything separately. Either way is fine.

Anaconda installation Page

Miniconda installation Page

Once Anaconda or Miniconda is installed do the following:

conda install Pillow requests numpy Shapely

Install the example project¶

To set up the example project, just use the –example flag when setting up a new project:

alg cp myproject --example

Installing documentation locally¶

If you want these docs to be installed locally, use the –with-docs flag when creating a project. You need to have Sphinx installed for this to work.

pip install Sphinx sphinx_rtd_theme
alg cp myproject --with-docs

Troubleshooting Install Issues¶

On some systems, additional libraries may be needed to install the Algorithm Toolkit. Try these packages if your ATK install fails:

Debian/Ubuntu Linux

# python 3
sudo apt install python3-dev build-essential

# python 2
sudo apt install python-dev build-essential

RedHat/CentOS Linux

# python 3
sudo yum install python3-devel

# python 2
sudo yum install python-devel

PyCharm Configuration for ATK¶

If you use PyCharm to develop Python applications, you can set it up to run the development environment for you rather than using the command line. The benefit of this is that you can then use the PyCharm debugging tools to set breakpoints and find out what’s going on with your application.

This page will explain how to set up PyCharm correctly. It assumes you know how to use the PyCharm debugger.

Adding a Configuration¶

Before you set up PyCharm, create your virtual environment and first project as described in Installation.

Once you’ve done this, in PyCharm select Edit Configurations from the Run menu. You’ll see a screen like this:

https://s3.amazonaws.com/atk-docs-images/pycharm-1.png

Click the “+” sign to Add New Configuration.

In the screenshot we’ve filled in the fields of information you’ll need:

Name: Give this configuration a name (like “Run ATK server”).
Single instance only: check this box.
Module name: when you first open this screen, the first field will read “Script path”. Click the small arrow next to the name to reveal the option “Module name”. Type the word flask into this field.

Note

As we mention elsewhere in the docs, each Algorithm Project is a Flask app.
Parameters: type the word run into this field.
Environment variables: there will already be a variable called PYTHONBUFFERED=1 in this field. Add a semicolon after that, and add: FLASK_APP=run;ATK_CONFIG= plus the full path to the config.py file in your Algorithm Project (e.g.: /home/myuser/atk/myproject/config.py). The whole line will read: PYTHONBUFFERED=1;FLASK_APP=run;ATK_CONFIG=/home/myuser/atk/myproject/config.py.

Note

On Windows, the path to the config.py file would be indicated more like this: C:/Users/myuser/atk/myproject/config.py. Note the use of forward slashes instead of the more typical back slashes.
Python interpreter: the version of Python inside your virtual environment. This is important, because the Python interpreter is linked to the environment containing all the project dependencies (like the Algorithm Toolkit itself). If you use a different interpreter PyCharm will not find the Python packages it needs.

Note

If you don’t see the interpreter for your virtual environment, select “Add Local” from the interpreter drop-down list. Then find the path to the python executable. On Mac OS and Linux, you can find it using which python. On Windows you may be able to use where python in the same way. NOTE that you must be in the virtual environment for this to work.
Working directory: the full path to your example project’s root folder.

Note

Unlike for the Environment variables field, on Windows this path will have back slashes instead of forward slashes.

Running the server¶

Once you’ve saved this configuration, you can run the server using the Play button next to the new configuration.

Algorithm Projects¶

When you use the Algorithm Toolkit (ATK), you will create and work on an Algorithm Project. A project contains one or more algorithms that you write, or that you install from the Algorithm Registry. It also contains some code to configure and run the project itself.

Creating a new project using the CLI will also create all the files and folders you need to get started. A new, empty project will look like this:

algorithms/
    __init__.py
chains/
logs/
    app.log
.env
.gitignore
__init__.py
config.py
licenses.json
run.py

Let’s look at what each of these files and folders do:

The algorithms/ folder will contain the algorithms used by this project. As you create or install algorithms, they will be placed in this folder.
The chains/ folder will contain chain definition files for each chain you create. As you create chains, their configuration files (in JSON format) will be placed in this folder.
The logs/ folder contains app.log, which algorithms can write to in order to provide information to a developer.
.env: Environment variables needed by the project are stored in this file.
.gitignore: This file will be familiar to you if you have used the git utility for managing source code. It tells git which files to leave out of a repository when saving or publishing it.
__init__.py: This file tells Python that the folder should be considered a Python package.
config.py: This is where you can set different parameters for the project, changing the way it works. See the Configuring a Project section below.
licenses.json: This file is a placeholder for now, but will store license keys issued by developers for algorithms that require them.
run.py: This file essentially links the project to the ATK.

Creating a project¶

The easiest way to create a project is to use the ATK CLI:

alg cp myproject

See the CLI docs for more information and examples.

Configuring a Project¶

config.py¶

The config.py file contains different project settings. A new project’s config.py will look like this:

import logging
import os
import sys

from logging import Formatter
from logging.handlers import RotatingFileHandler


SECRET_KEY = os.environ['FLASK_SECRET_KEY']
ATK_PATH = os.path.dirname(os.path.abspath(__file__))
API_KEY = os.environ['ATK_API_KEY']

sys.path.append(ATK_PATH)
dirname = os.path.dirname
logfile = os.path.join(dirname(__file__), 'logs', 'app.log')
handler = RotatingFileHandler(logfile, maxBytes=10240, backupCount=10)
handler.setLevel(logging.DEBUG)
handler.setFormatter(Formatter(
    '%(asctime)s %(levelname)s %(message)s [in %(pathname)s:%(lineno)d]'
))
# Add additional logger handlers here, add to LOG_HANDLERS list

LOG_HANDLERS = [handler, ]

# Add your custom settings below

These settings should probably not be changed, but you can add your own as the comment at the bottom of the file indicates.

About logging¶

As you can see from the configuration here, the LogLevel by default is set to DEBUG. If you want to write more information to your log file (located in logs/app.log in your project), then keep this at DEBUG. If you want to write less - such as when you’re in a production setting - you can raise this to something like WARNING or ERROR. Just change it like so:

handler.setLevel(logging.WARNING)

See Python’s documentation on logging for more details.

Optional settings¶

Here are some additional settings you can add to the project using config.py:

CORS_ORIGIN_WHITELIST¶

You probably will never need to adjust this setting, but if you create an application that uses the ATK (e.g.: to run a chain from another application on the web), then that application’s URL needs to be in this list. This is a safety measure to prevent unwanted apps from hitting your site.

Default: []

Example:

CORS_ORIGIN_WHITELIST = ['http://mysite.com', ]

DEFAULT_WORKING_ROOT¶

When a chain runs, an algorithm can use this folder to store temporary files as well as file-based results. Each chain gets a unique ID, and the ATK makes a folder under DEFAULT_WORKING_ROOT with that ID as its name. Within that folder, a folder named temp and one named results are also created. Anything stored in temp gets deleted after a chain finishes, but files in results remain and can be used later.

Default: '/tmp'

Example:

DEFAULT_WORKING_ROOT = '/users/myusername/atk'

.env¶

.env is a special configuration file. Anything placed in this file gets added to the system environment variables when the ATK is running. This is a place to store information that will be different from one environment to another (e.g.: one set of variables for development, one for production). Because of this, the .env file is listed in .gitignore and will not be added to a git repository if you create one. It’s also a place to store information you don’t want anyone to see (like security tokens).

A new project will have a .env file that looks like this:

FLASK_SECRET_KEY="*********************************************************************"
ATK_API_KEY="********************"
ATK_MANAGEMENT_API_KEY="****************************************"
FLASK_ENV=development

The “*”s will be random characters generated when you create the project using the CLI.

Notice that this is not a Python program, but just a text file. Here is an explanation of the parameters:

FLASK_SECRET_KEY¶

This is a key used internally by Flask to protect data submitted through forms and also to sign cookies. This key can contain unicode characters.

To be on the safe side, do not change this value

ATK_API_KEY¶: This key is used when running a chain. You will paste it into the Test Run form when testing your chains, and use it when you run a chain from an external program. It’s also used when querying the ATK about what chains and algorithms are installed.

ATK_MANAGEMENT_API_KEY¶

This key is used to find out information about the ATK node, like how much load the system has or to retrieve the application log file. This key must be different than the ATK_API_KEY. If you don’t want to enable these features, you can remove this line from the .env file.

Note: If you use TileDriver Process™, removing this key will reduce functionality

FLASK_ENV¶

This is another Flask internal configuration setting. The two recognized options are “development” and “production”. The development environment enables “DEBUG” in Flask automatically, which provides you the developer with useful information when testing out your code.

Also, the development environment itself cannot be accessed when this is set to “production”.

Note

You do not use quotation marks for this setting. The line would be:

FLASK_ENV=production

if you wanted to use the production environment.

The ATK Development Environment¶

When you work on algorithm projects, you can use the ATK’s web-based development environment to make your life easier.

Running the environment¶

Once you have created a project, you can use:

alg run

to start the development environment. This command starts a basic web server and provides access to API endpoints within the ATK that allow you to create and edit algorithms and processing chains, and test your chains to see if they work.

Note

If you are running the ATK virtual server, such as with VirtualBox, you may need to run the development environment with the –host parameter as follows:

alg run --host=0.0.0.0

After you use alg run, you can point your web browser to http://localhost:5000/ and you will see the screen below:

https://s3.amazonaws.com/atk-docs-images/dev-environment-home.png

Under the hood¶

When you use alg run from a Terminal window, you will see lines like the following appear:

* Serving Flask app "run" (lazy loading)
* Environment: development
* Debug mode: on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 225-327-370

If you’ve worked with the web microframework Flask, this will look familiar. Each algorithm project is a Flask app.

Here’s what’s happening:

The name of the app is “run” (the run.py module in your project)

You’re in “development” mode

“Debug” is enabled

The address to which to point your browser is http://127.0.0.1:5000/ (you can also use localhost, which is easier to remember)

It also gives you a debugger PIN code, which will be handy later (see the Debugging section)

Stopping The Environment¶

From the terminal window. you can hit ^C on your keyboard to stop the development server. On the web interface, the home page has a Stop button in the footer (see screen shot above); this can be useful when it seems as though your development server will not shut down.

The Project Homepage¶

The first page you come to (shown above) is the project homepage (naturally enough). This gives you some helpful hints and links about what you can do with the ATK. There are also other links to related sites you may find useful.

Across the top of the screen is the ATK menu bar. It provides access to these areas of the ATK development environment:

Project Home (the current page)

Algorithms: A listing of the algorithms installed in this project, and information about them

Chain Builder: A tool to link algorithms together in processing chains

Documentation: A link to this site, or to local docs if you created the project using the –with-docs flag

Test Run: A test harness for running algorithm chains

Let’s go over each of these pages in turn.

Algorithms Page¶

For a new project, this page will be empty except for a link to create an algorithm. If you installed the example project, you’ll see this screen:

https://s3.amazonaws.com/atk-docs-images/dev-environment-algorithms.png

You can click on the name of an algorithm to get more information about it.

https://s3.amazonaws.com/atk-docs-images/dev-environment-algorithms-expanded.png

See the section Working With Algorithms for more details about what this page does.

Chain Builder Page¶

If you installed the example project, you’ll see this page:

https://s3.amazonaws.com/atk-docs-images/dev-environment-chain-builder.png

The example project comes with one chain, called “map_tiles”. You can select it from the drop-down menu that reads “Select a chain to view/edit”. When you select a chain, the chain definition is displayed:

https://s3.amazonaws.com/atk-docs-images/dev-environment-chain-builder-show-chain.png

See the section Building Chains for more details on this feature.

Test Run Page¶

Each chain you create becomes an option under the “Test Run” menu in the top navigation bar. In the example project, selecting the “map_tiles” chain displays this screen:

https://s3.amazonaws.com/atk-docs-images/dev-environment-test-run.png

The form on this page allows you to run the chain by inputting various parameters and clicking “Run Algorithm Chain”. See the section Testing Algorithm Chains for more details on how to use the test harness.

Working With Algorithms¶

What is an algorithm?¶

In terms of the ATK, an algorithm is a Python package that contains at least some Python code that follows a specific structure, and a JSON file describing the algorithm’s input and outputs. Like any Python package, an algorithm can contain a lot of other things as well.

Creating an Algorithm¶

There are three ways you can create an algorithm:

Use the development environment web interface (see below section)

Use the CLI (see the CLI section in the docs for the ca command)

Manually create it yourself

We recommend the first two options over the third. We’ve created a lot of checks to make sure things like typos and missing files don’t happen.

Structure Of An Algorithm¶

When you create an algorithm using the CLI or the development environment web interface, the following five files will be created inside the algorithms/algorithm_name folder in your project:

__init__.py
algorithm.json
LICENSE
main.py
README.md
test.py

Let’s look at each of these in turn.

__init__.py: This file tells Python to consider this folder a Python package; it will probably never have anything in it

algorithm.json: This file contains the algorithm’s configuration, including any input parameters or data that the algorithm outputs

LICENSE: This file contains whatever licensing terms you want to place on the algorithm

main.py: This is where you will write the code that your algorithm runs, or call code you write in other Python programs

README.md: A helpful document for users of your algorithm

test.py: A module for your algorithm’s unit tests

Most of the time you will edit only main.py (and probably test.py) directly. The development environment web interface handles updates to the other files for you.

algorithm.json¶

Even though the web interface updates this file on your behalf as you change your algorithm, you can edit it yourself if you wish. Here’s what one looks like:

{
        "name": "getmaptiles_roi",
        "display_name": "Get Map Tiles In ROI",
        "description": "This algorithm will gather up map tiles at a given zoom level that intesect with the provided polygon. The source is the national map provided by USGS. All tiles will be written out to disk at a specified location. This location is also saved onto the chain ledger.",
        "version": "0.0.1",
        "license": "MIT",
        "private": false,
        "homepage": "https://tiledriver.com/developer",
        "required_parameters": [
                {
                        "name": "roi",
                        "description": "Polygon WKT to obtain tiles that intersect.",
                        "display_name": "Polygon WKT",
                        "data_type": "string",
                        "field_type": "text",
                        "help_text": "Enter well known text (WKT) for the polygon region to obtain intersecting tiles.",
                        "min_value": null,
                        "max_value": null,
                        "default_value": "POLYGON((-77.0419692993164 38.9933585922412,-77.17311859130861 38.891887936025896,-77.03853607177736 38.790272111428706,-76.91013336181642 38.891887936025896,-77.0419692993164 38.9933585922412))",
                        "custom_validation": null,
                        "parameter_choices": [],
                        "sort_order": 0
                },
                {
                        "name": "zoom",
                        "description": "Zoom level",
                        "display_name": "Zoom level",
                        "data_type": "integer",
                        "field_type": "number",
                        "help_text": "Enter an integer corresponding to the zoom level, 16 is max value supported.",
                        "min_value": 5,
                        "max_value": 16,
                        "default_value": 14,
                        "custom_validation": null,
                        "parameter_choices": [],
                        "sort_order": 1
                },
                {
                        "name": "cache_path",
                        "description": "Path to save gathered tiles to.",
                        "display_name": "Tile Cache Location",
                        "data_type": "string",
                        "field_type": "text",
                        "help_text": "Absolute path to directory for saving tiles",
                        "min_value": null,
                        "max_value": null,
                        "default_value": "/tmp/tiles/map_tiles",
                        "custom_validation": null,
                        "parameter_choices": [],
                        "sort_order": 2
                }
        ],
        "optional_parameters": [],
        "outputs": [
                {
                        "name": "image_chips_dir",
                        "description": "Local path to directory of map tiles",
                        "display_name": "Path To Tiles",
                        "data_type": "string"
                }
        ]
}

Here’s what each of the fields means:

Parameter	Description
name	A short name with no spaces, which must be unique across algorithms in your project
display_name	A more descriptive name that will be displayed to the user
description	A long description of what the algorithm does
version	The algorithm version number in text form
license	A description of the license terms, using the SPDX identifier (https://spdx.org/licenses/)
private	If the algorithm is published to the Algorithm Registry, setting private to true will cause it cause it not to display in a Registry listing
homepage	A web address where more information can be found about the algorithm (could also be a link to a source code repository)

The algorithm.json file also defines input and output parameters. For inputs, there are “required_parameters” and “optional_parameters”. The list of outputs is just called “outputs”. Each parameter and each output has the same structure:

Input Parameters

Parameter	Description
name	A short name with no spaces, which must be unique across parameters in this algorithm
display_name	A more descriptive name that will be displayed to the user
description	A long description of what the parameter is for
data_type	The type of data the input parameter will accept
field_type	When displayed on a web form, which HTML field type to use
help_text	Text that would be displayed below a form field, providing guidance on how to enter a value
min_value	If a numeric field, what is the minimum value the input should accept
max_value	If a numeric field, what is the maximum value the input should accept
default_value	The value to use if not provided (also will display in a web form)
custom_validation	Custom validation rules (see below)
parameter_choices	A comma-separated list of values that the field will accept (often goes with a Select field type, but does not have to)
sort_order	On a web form, where should this field be displayed relative to other input fields

Outputs

Parameter	Description
name	A short name with no spaces, which must be unique across outputs of this algorithm
display_name	A more descriptive name that will be displayed to the user
description	A long description of what the output is
data_type	The type of data the output will produce

It may seem like a lot of information for you to fill out, but the more detail you enter into algorithm.json the easier it will be for another developer to use what you create. An important part of the ATK is code sharing and reuse among the developer community.

main.py¶

As stated earlier, most of your code will live in this file, or this file will point to other code you have written. When you create a new algorithm, the main.py will look like this:

from algorithm_toolkit import Algorithm, AlgorithmChain


class Main(Algorithm):

    def run(self):
        cl = self.cl  # type: AlgorithmChain.ChainLedger
        params = self.params  # type: dict
        # Add your algorithm code here

        # Do not edit below this line
        return cl

It may not look like much, but there’s a lot going on in these few lines of code.

Every time your algorithm runs, this run() routine will be called. As you’ll notice, this Main class inherits from a class inside the ATK called Algorithm. To see what the Algorithm class can do, check out the API section of these docs.

When this code runs, a dictionary called params will contain the data entered by a user or application calling the algorithm. In the get_map_tiles example above, params would look like this:

{
    'roi': 'POLYGON((-77.0419692993164 38.9933585922412,-77.17311859130861 38.891887936025896,-77.03853607177736 38.790272111428706,-76.91013336181642 38.891887936025896,-77.0419692993164 38.9933585922412))',
    'cache_path': '/tmp/tiles/map_tiles',
    'zoom': 14
}

If you wanted to get the value of the zoom parameter in your code, you could refer to it like this:

zoom = params['zoom']

This is how your algorithm can receive the data it needs to operate. Notice that the zoom parameter is an integer in the params dictionary? The ATK does some nice things for you, such as making sure the data will be in the proper format you need it to be in (via the data_type field in the input parameter). If it’s not, the user will be alerted to the problem and your code won’t even run. This means you can focus on making use of the data inputs rather then spending time validating them.

You also probably noticed a reference to a mysterious variable called simply cl. This stands for Chain Ledger, and is one of the most important aspects of the ATK. It’s so important, it has its own section in the docs.

The short version is: the Chain Ledger is how you will provide outputs to other algorithms in the chain, and how you can see a history of everything the chain did after it runs. This is a hugely powerful and useful feature, because it gives you the ability to reproduce results at a later time.

You can put literally anything Python can create on the Chain Ledger. Most of the time, you will just place your algorithm’s outputs on it. In the get_map_tiles example, the algorithm might do it like so:

cl.add_to_metadata('image_chips_dir', '/tmp/chips')

See the Chain Ledger section for more details on how this works.

Algorithm Input Validation¶

Data types¶

The data_type field can be set to any one of the following options:

String (the default)

Integer

Float

Array (entered as a comma-separated list in a form field; the list members can be any data type)

Before an algorithm is called, the ATK uses the data_type to ensure that the correct type is being sent to the algorithm. If the value can be cast to the correct type, the ATK will do that. Otherwise an error will be raised.

Validation controls¶

We said earlier that the ATK validates inputs before your code runs. In addition to the data type validation, you can set boundaries on what values will be accepted. Several fields within the parameter definition help with this: min_value, max_value, parameter_choices, and custom_validation. The first three are self-explanatory; let’s look at custom_validation.

custom_validation¶

The ATK comes with a set of validators you can refer to in this field. They are:

greaterthan¶

The value must be greater than the value of another input parameter. Numeric inputs only.

Example:

"custom_validation": "greaterthan.another_input"

where “another_input” is the name of the parameter whose value that must be smaller than this value.

Note

greaterthan and lessthan should come in pairs. In other words, if input1 should be greater than input2, input1 should use greaterthen.input2 as its custom validation, and input2 should have lessthan.input1 as its custom validation.

lessthan¶

The value must be less than the value of another input parameter. Numeric inputs only.

Example:

"custom_validation": "lessthan.another_input"

where “another_input” is the name of the parameter whose value that must be greater than this value.

Note

greaterthan and lessthan should come in pairs. In other words, if input1 should be less than input2, input1 should use lessthen.input2 as its custom validation, and input2 should have greaterthan.input1 as its custom validation.

evenonly¶

The value must be an even integer. The data_type must also be Integer.

Example:

"custom_validation": "evenonly"

oddonly¶

The value must be an odd integer. The data_type must also be Integer.

Example:

"custom_validation": "oddonly"

^

The value must conform to a regular expression pattern, defined after the caret (^) symbol. String inputs only.

Example:

"custom_validation": "^\d{3}-\d{3}-\d{4}$"

This would require the input to have a US phone number-like format. See the Python documentation for regular expression syntax.

Outputs¶

If your algorithm outputs a value, that value must also have a data type. When building chains, it’s important to think about what type of output your algorithm produces because otherwise the next algorithm in the chain might not get a value of the right data type (and thus would never run, as we stated earlier). In fact, the Chain Builder prevents you from linking an output of one algorithm to the input of another if they’re not the same data type (more on this in the Chain Builder section).

Using The Web Interface¶

When you click the Create a New Algorithm button on the Algorithms page, you will see the create/edit algorithm form:

https://s3.amazonaws.com/atk-docs-images/working-with-algorithms-form.png

If you pull up the get_map_tiles algorithm, you can see the form filled out with the data from its algorithm.json:

https://s3.amazonaws.com/atk-docs-images/working-with-algorithms-form-filled.png

Only the algorithm name and display name fields are required, but the more information you provide the easier it will be for another developer to use your algorithm.

License¶

We provide several open source licenses that can be applied to your algorithm. When you click the license drop-down menu, you’ll see the following options:

When you choose a license, a LICENSE file will be placed in your algorithm’s folder corresponding to the open source license version you select. The text of these licenses come from the Open Source Initiative website (https://opensource.org/licenses).

If you do not wish to apply an open source license, you can choose one of two additional options: “Proprietary” or “See LICENSE File”. Both of these options will create a blank LICENSE file in your algorithm’s folder, and you can add whatever terms you deem appropriate.

Adding and editing parameters¶

If you click the Edit button (looks like a pencil) for the roi parameter, you will see the create/edit input parameter form slide in from the right:

https://s3.amazonaws.com/atk-docs-images/working-with-algorithms-form-parameter.png

Numeric parameters (integers and floats) look like this:

https://s3.amazonaws.com/atk-docs-images/working-with-algorithms-form-integer-parameter.png

Adding and editing Outputs¶

If you click the Edit button (looks like a pencil) for the image_chips_dir output, you will see the create/edit output form slide in from the right:

https://s3.amazonaws.com/atk-docs-images/working-with-algorithms-form-output.png

Head over to Creating Your First Algorithm to see this form in action.

Creating Your First Algorithm¶

Let’s dive into algorithm development with everyone’s favorite trope: the Hello World program!

We’ll use the development environment web interface to set up the algorithm, then modify the code in main.py to enhance the functionality of our algorithm.

Creating a Project¶

We’ll need an algorithm project to start things off. Take a look at the Installation documentation for how to create a new project with the example project included. Having the example available is a great reference for how to make your own algorithms.

Setting Things Up¶

As you’ll recall from the last page, the create new algorithm form looks like this (click the Create a New Algorithm button from the Algorithms page):

Let’s fill in some details about our algorithm. We’ll name it “my_first_algorithm”, give it a brief description, and choose the MIT license (we’re feeling generous today):

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-1.png

Click “Save Algorithm” and you’ll see your algorithm appear in the list:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-2.png

That’s it! Notice we did not even have to name any inputs or outputs in order to create the algorithm. It will be more useful once we have those, but for now let’s leave it at that.

What Just Happened?¶

After we clicked the Save button, the ATK created a new folder in our project called algorithms/my_first_algorithm. Within that folder, you’ll find the same files described in Working With Algorithms:

__init__.py
algorithm.json
LICENSE
main.py
README.md
test.py

The algorithm.json file contains the basic information you entered:

{
    "description": "This is my first algorithm, and I'm proud of it.",
    "display_name": "My First Algorithm",
    "homepage": "",
    "license": "MIT",
    "name": "my_first_algorithm",
    "optional_parameters": [],
    "outputs": [],
    "private": false,
    "required_parameters": [],
    "version": "0.0.1"
}

Now What?¶

We have the basic structure of an algorithm; now we need to make it do something.

Open up the main.py file in your new algorithm folder, using your favorite text editor or programming tool. You’ll see the following code:

from algorithm_toolkit import Algorithm, AlgorithmChain


class Main(Algorithm):

    def run(self):
        cl = self.cl  # type: AlgorithmChain.ChainLedger
        params = self.params  # type: dict
        # Add your algorithm code here

        # Do not edit below this line
        return cl

Working With Algorithms describes the basics of what’s going on here.

In order to make our Hello World program complete, we need to have it greet the world with its friendly message. To send a message to the user or calling function, we need to make use of the Chain Ledger. You’ll recall from Working With Algorithms that the Chain Ledger is a Python dictionary, and you can access it using the variable cl in your algorithm.

We’ll use a special key the Chain Ledger maintains: chain_output_value. Essentially, this key outputs a message or other result to the client. The output value can be unstructured text, JSON, CSV, or even binary files. For our program we’ll just use text. Here’s how you would output the famous greeting in your algorithm:

# Add your algorithm code here

chain_output = {
    'output_type': 'text',
    'output_value': 'Hello World!'
}
cl.add_to_metadata('chain_output_value', chain_output)

# Do not edit below this line
return cl

As you can see, chain_output_value is itself a dictionary with two keys: output_type (“text” in this case) and output_value (“Hello World!”). You use the Chain Ledger’s add_to_metadata function to set the value of this key, and the function takes two arguments: the key name (“chain_output_value”) and the key value (our dictionary).

Save the main.py file.

Running The Algorithm¶

In order to run this code, we need to set up a basic chain. The ATK does not run individual algorithms: it requires a chain, and once you create one you can use the Test Run harness provided in the development environment.

In the web interface, click the link for Chain Builder. You’ll see your algorithm listed along with the example project algorithms:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-3.png

When you click the name of your algorithm, you’ll see a block representing it:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-4.png

Drag the block to the right onto the Chain Canvas. You’ll get a prompt to give this chain a name:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-5.png

After that you’ll see your chain, with your single algorithm:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-6.png

Believe it or not, that’s it! However modest, you now have an algorithm processing chain.

The chain name will now appear in the Test Run menu at the top of the screen. Select it, and you’ll see a basic web form:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-7.png

Because we don’t have any input parameters, there’s not much here. You will however need your API key.

What is my API key?¶

The API key is a simple security mechanism to make sure no one runs your chain or does anything with your algorithm project unless you want them to. To find your key, open the .env file in your algorithm project, and copy the value in ATK_API_KEY (don’t copy the quotation marks).

Paste the value into the API Key form field. Then click “Run Algorithm Chain”.

This chain will run pretty fast, since you’re just outputting text to the screen. After it’s finished, you’ll see this:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-8.png

There it is! Our welcome message to the world.

Adding An Input¶

Now that we have a working algorithm and chain, we begin to see all kinds of possibilities. Let’s start by allowing our greeting to be more personal.

Edit your algorithm in the web interface by clicking the Edit button next to its name in the Algorithms page. Then click Add Parameter.

We’ll add a required input for the name of the person we want to greet. Fill out the relevant fields like this:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-9.png

Click the Save button on the parameter form. You’ll see the parameter listed:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-10.png

Now click Save Algorithm.

If you open your algorithm.json file, you’ll see the new parameter has been added to the definition:

{
    "description": "This is my first algorithm, and I'm proud of it.",
    "display_name": "My First Algorithm",
    "homepage": "",
    "license": "MIT",
    "name": "my_first_algorithm",
    "optional_parameters": [],
    "outputs": [],
    "private": false,
    "required_parameters": [
        {
            "custom_validation": "",
            "data_type": "string",
            "default_value": "",
            "description": "What should I call you?",
            "display_name": "Name of person to greet",
            "field_type": "text",
            "help_text": "Please enter your full name",
            "max_value": "",
            "min_value": "",
            "name": "greeting_name",
            "parameter_choices": "",
            "required": true,
            "sort_order": 0
        }
    ],
    "version": "0.0.1"
}

Since we added an input parameter, we need to modify the chain to tell the ATK how the input will get a value. Click Chain Builder, then select your chain from the drop-down menu:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-11.png

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-12.png

Look at that! The ATK figured out that you wanted the user to input a value for the new field without you having to do anything. Awesome!

There are other options for how an input gets its value, which we’ll explore in a later section. For now, let’s run this chain using Test Run again:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-13.png

You’ll see the new input parameter on the form, along with the help text explaining what the field is for. If you hover over the info icon to the right, you’ll see the description you entered for the parameter:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-14.png

Click Run Algorithm Chain.

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-15.png

Huh, you got the same result. How come?

Using The Input¶

Your algorithm code needs to do something with the input value. Let’s open main.py again and change the code like this:

# Add your algorithm code here

name = params['greeting_name']
msg = 'Hello, %s!' % name

chain_output = {
    'output_type': 'text',
    'output_value': msg
}
cl.add_to_metadata('chain_output_value', chain_output)

# Do not edit below this line
return cl

Here you see we’re using the params dictionary to take the parameter called “greeting_name” and assign it to a variable. The ATK saves the user input in that key in params. While you’re developing, it can be helpful to open the algorithm.json file if you forget what you named a parameter.

Then we create the new greeting and assign it to the “output_value” for placing on the Chain Ledger. Looks like we’re done here.

Save the changes to main.py

Note

The development environment can detect some changes to your algorithm code and restart itself so you don’t have to remember to do it. When that happens, you’ll see lines like this in your Terminal window:

* Detected change in '/myproject/algorithms/my_first_algorithm/main.py', reloading
* Restarting with stat
* Debugger is active!
* Debugger PIN: 225-327-370

Now go back to Test Run and run your chain. This time you’ll see the right greeting:

https://s3.amazonaws.com/atk-docs-images/creating-your-first-algorithm-16.png

This is fantastic. You can create algorithms that accept user input, process that input, and display something to the user. In all, you wrote six lines of code and filled out a couple of forms. Not bad.

Where To Go From Here?¶

Although this was a basic example, you’ve seen how most of the development environment works and how you can use the ATK to run an application. You’re ready to take on a bigger project.

We suggest you go through the next section Working With Chains to understand how chains work and how to manipulate them. Then, you can run through the Tutorial: Do Maths to see how you can practically use processing chains. You’ll also learn how to log information and provide status updates to the user.

Working With Chains¶

Now you’ve seen how to create a single algorithm and make it do something. Pretty cool, but the power of the ATK really shines with processing chains.

What Is A Processing Chain?¶

Simply put, a processing chain in the ATK is a series of algorithms, each linking to the next via output parameters.

Imagine a chain like a set of tasks. When you mail a package to someone, your task is complete when you drop it off at the shipping company. But to get to the recipient the package has to go through a series of waystations, each with its own tasks. The whole chain might look like this:

You drop off your package at the post office in the small town you live in

The post office sends it to a central office somewhere else in the state

The state hub sends the package to another hub

That hub sends the package to a local office near the recipient

A local delivery person drops it off at the recipient’s location

At each step, information is read about where the package goes next. In addition, the final destination address has to be carried along the route so it ends up in the right place.

Think of the next destination along the route as an input to an algorithm, the journey between points as algorithms, and the information contained in the overall route as the Chain Ledger, and you’ve basically got the idea. After the package is delivered, you could do back and see all the steps the package went through, how long it took to reach each point, and so on.

How Does This Actually Work?¶

You define algorithms just like you did in the last section Creating Your First Algorithm. Then, you link those algorithms together to create a workflow, or processing chain. Then you can test the chain using our test harness.

Examining The Example Project¶

Sometimes the best way to learn is to read working code. The example project comes with three algorithms and a processing chain; let’s examine those.

Go to the Chain Builder in the web interface and select the “map_tiles” chain. This is what it looks like:

The left side of this screen contains the algorithms installed in your project. To the right of that and taking up most of the screen, is the Chain Builder Canvas. As you can see, “map_tiles” links the algorithms together in the following order (top to bottom):

Get Map Tiles in ROI

Stitch together tiles

Output Image to Web

You’ll also notice that each algorithm is getting its data from different sources. The three inputs for “Get Map Tiles in ROI” come from the user, but the other two algorithms get data from algorithms that came before. “Stitch together tiles” gets its one input value from an output of “Get Map Tiles in ROI” called “image_chips_dir”. “Stitch together tiles” also provides two outputs for “Output Image to Web”: “image_path” and “image_bounds”.

Let’s take a step back and talk about what this chain actually does. In the first algorithm, this chain acquires map tiles from a public tile server (provided by the U.S. Geological Survey) for a given region of interest (ROI). The chain then hands those tiles over to a second algorithm, which stitches the image tiles together into a GeoTIFF mosaic that is orthorectified. Finally, the chain hands the mosaic to a third algorithm that converts the GeoTIFF to a web format (PNG) and sends the PNG data along with the geographic bounding box data back to the client.

To use this chain, it’s not really necessary for you to know how the individual algorithms do what they do. All you need to know is what information goes in, and what comes out. However, because you have the source code for the algorithms in the chain, you can see how it all works.

That’s the beauty of the ATK platform: you can use it to do some really complicated things and have it “just work,” and you can also see the details about how the work was done.

We built the ATK specifically to have these two capabilities. Many who use the ATK rely on its power to make their data processing lives easier, while others need to tweak the algorithms in order to understand fully the results of a chain run.

Chain Definition Files¶

The Chain Builder provides a graphical view for a chain’s structure and lets you change how it’s put together. Under the hood, the chain is defined by JSON files in your project called chains/<CHAIN_NAME>.json. Here’s what one looks like with the example project:

{
    "map_tiles": [
        {
            "algorithm": "getmaptiles_roi",
            "parameter_source": "user"
        },
        {
            "parameters": {
                "image_chips_dir": {
                    "source": "chain_ledger",
                    "occurrence": "first",
                    "key": "image_chips_dir",
                    "source_algorithm": "getmaptiles_roi"
                }
            },
            "algorithm": "stitch_tiles"
        },
        {
            "parameters": {
                "image_bounds": {
                    "source": "chain_ledger",
                    "occurrence": "first",
                    "key": "image_bounds",
                    "source_algorithm": "stitch_tiles"
                },
                "image_path": {
                    "source": "chain_ledger",
                    "occurrence": "first",
                    "key": "image_path",
                    "source_algorithm": "stitch_tiles"
                }
            },
            "algorithm": "output_image_to_client"
        }
    ]
}

This JSON file would be called chains/map_tiles.json. You can see the same structure depicted in graphical form here:

“getmaptiles_roi” gets its input from “user”

“stitch_tiles” its one input from “getmaptiles_roi”

“output_image_to_client” gets both of its inputs from “stitch_tiles”

If an algorithm gets all of its inputs from the user, you will see “parameter_source”: “user”. Otherwise, the next link in the processing chain will have a “parameters” dictionary applied, with each parameter named as a key: value pair. If the value comes from the user, it will read:

{
    "source": "user"
}

Otherwise, it will have the additional fields as shown here:

{
    "source": "chain_ledger",
    "occurrence": "first",
    "key": "",                 <-- name of parameter in source algorithm
    "source_algorithm": ""     <-- name of the source algorithm
}

What is ‘occurrence’?¶

Some processing chains use the same algorithm more than once. In these cases, the “occurrence” flag is a mechanism to tell the ATK which occurrence of the algorithm to use for this input. The Chain Builder algorithm blocks have a small drop-down menu for the chain developer to use to indicate this value.

The Chain In Action¶

When you pull up a chain in the Test Run page, you will see its web form:

What you see here is probably making more sense now. This view is presented by the ATK as an interface to the processing chain. In fact, the data to run a chain does not have to come from a web form at all; however, it’s a convenient way to explain how this all works.

Copy and paste your API Key into the field provided. You’ll notice that all the other fields are filled out: those are the default values defined by the algorithm creator. Especially in the case of the ROI field, this can really make a user’s life easier. Let’s keep these defaults and click “Run Algorithm Chain”.

https://s3.amazonaws.com/atk-docs-images/working-with-chains.png

When the chain runs, you’ll see some status information and a progress bar indicating how far along the chain we are. An algorithm developer can also pass algorithm progress to the user as we’ll see in another section.

In this case, the final output to the client appears in a web map. How did this happen?

Chain outputs¶

In the section Creating Your First Algorithm, you output text to the test client. The ATK allows you to output a variety of other formats:

geo_raster¶

Send a PNG or JPG to the client along with the geographic bounds (in lat,lon pairs) of the image. The image will display in a web map.