# Testing
All unit and lint tests are run using `tox`, with different versions of Python with specific constraints. We also use `tox-setuptools-version` to specify the correct setuptools version based on what is present in each release, and `tox-pyenv` to recognize the different local [pyenv interpreters](https://github.com/pyenv/pyenv).

First, run the script to install and configure `pyenv`, and the `tox` dependencies:

```shell
./tools/setup_pyenv.sh
```

After that you need to [set up your shell environment](https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv), according to the pyenv documentation.
The guide has quick snippets to configure `bash`, `zsh` and `fish` shells.

Refresh your terminal to make sure pyenv is working. Then you can run the unit and lint tests:

```shell
tox
```

> **Note**
> There are a number of `autouse` mocks in our unit tests. These are intended to prevent accidental side effects on the host system from running the unit tests, as well as prevent leaks of the system environment into the unit tests.
> One such `autouse` mock tells the unit tests that they are run as root (unless the mock is overriden for a particular test).
> These `autouse` mocks have helped, but may not be preventing all side effects or environment leakage.

The client also includes built-in dep8 tests. These are run as follows:

```shell
autopkgtest -U --shell-fail . -- lxd ubuntu:xenial
```

## Integration Tests

Ubuntu Pro Client uses [behave](https://behave.readthedocs.io)
for its integration testing.

The integration test definitions are stored in the `features/`
directory and consist of two parts: `.feature` files that define the
tests we want to run, and `.py` files which implement the underlying
logic for those tests.

By default, integration tests will do the following on a given cloud platform:
 * Launch an instance running latest daily image of the target Ubuntu release
 * Add the Ubuntu Pro client daily build PPA: [ppa:ua-client/daily](https://code.launchpad.net/~ua-client/+archive/ubuntu/daily)
 * Install the appropriate ubuntu-advantage-tools and ubuntu-advantage-pro deb
 * Run the integration tests on that instance.

The testing can be overridden to install ubuntu-advantage-tools from other sources instead of the daily PPA by providing `UACLIENT_BEHAVE_INSTALL_FROM` to the behave test runner. The default is `UACLIENT_BEHAVE_INSTALL_FROM=daily`, and the other available options are:

- `staging`: install from the [staging PPA](https://code.launchpad.net/~ua-client/+archive/ubuntu/staging)
- `stable`: install from the [stable PPA](https://code.launchpad.net/~ua-client/+archive/ubuntu/stable)
- `archive`: install the latest version available in the archive, not adding any PPA
- `proposed`: install the package from the -proposed pocket - specially useful for SRU testing (see [the release guide](how_to_release_a_new_version_of_ua.md))
- `custom`: install from a custom provided PPA. If set, then two other variables need to be set: `UACLIENT_BEHAVE_CUSTOM_PPA=<PPA URL>` and `UACLIENT_BEHAVE_CUSTOM_PPA_KEYID=<signing key for the PPA>`.
- `local`: install from a local copy of the ubuntu-advantage-client source code

`local` is particularly useful, as it runs the suite against the local code, thus including and validating the latest changes made. It is advised to run any related integration tests against local code changes before pushing them to be reviewed.

> **Note**
> Note that we cache the source when running with `UACLIENT_BEHAVE_INSTALL_FROM=local` based on a hash, calculated from the repository state. If you change the python code locally and run the behave tests against your new version, there will be new debs in the cache source with the new repo state hash.

To run the tests, you can use `tox`:

```shell
tox -e behave-lxd-20.04
```

or, if you just want to run a specific file, or a test within a file:

```shell
tox -e behave-lxd-20.04 features/unattached_commands.feature
tox -e behave-lxd-20.04 features/unattached_commands.feature:28
```

As can be seen, this will run behave tests only for release 20.04 (Focal Fossa). We are currently
supporting 5 distinct releases:

* 21.10 (Kinetic Kudu)
* 22.04 (Jammy Jellyfish)
* 20.04 (Focal Fossa)
* 18.04 (Bionic Beaver)
* 16.04 (Xenial Xerus)

Therefore, to change which release to run the behave tests against, just change the release version
on the behave command.

Furthermore, when developing/debugging a new scenario:

 1. Add a `@wip` tag decorator on the scenario
 2. To only run @wip scenarios run: `tox -e behave-lxd-20.04 -- -w`
 3. If you want to use a debugger:
    1. Add ipdb to integration-requirements.txt
    2. Add ipdb.set_trace() in the code block you wish to debug

(If you're getting started with behave, we recommend at least reading
through [the behave
tutorial](https://behave.readthedocs.io/en/latest/tutorial.html) to get
an idea of how it works, and how tests are written.)

## Iterating Locally

To make running the tests repeatedly less time-intensive, our behave
testing setup has support for reusing images between runs via two
configuration options (provided in environment variables),
`UACLIENT_BEHAVE_IMAGE_CLEAN` and `UACLIENT_BEHAVE_REUSE_IMAGE`.

To avoid the test framework cleaning up the image it creates, you can
run it like this:

```sh
UACLIENT_BEHAVE_IMAGE_CLEAN=0 tox -e behave
```

which will emit a line like this above the test summary:

```
Image cleanup disabled, not deleting: behave-image-1572443113978755
```

You can then reuse that image by plugging its name into your next test
run, like so:

```sh
UACLIENT_BEHAVE_REUSE_IMAGE=behave-image-1572443113978755 tox -e behave
```

If you've done this correctly, you should see something like
`reuse_image = behave-image-1572443113978755` in the "Config options"
output, and test execution should start immediately (without the usual
image build step).

(Note that this handling is specific to our behave tests as it's
performed in `features/environment.py`, so don't expect to find
documentation about it outside of this codebase.)

## Optimizing total run time of integration tests with snapshots
When `UACLIENT_BEHAVE_SNAPSHOT_STRATEGY=1` we create a snapshot of an instance
with ubuntu-advantage-tools installed and restore from that snapshot for all tests.
This adds an upfront cost that is amortized across several test scenarios.

Based on some rough testing in July 2021, these are the situations
when you should set UACLIENT_BEHAVE_SNAPSHOT_STRATEGY=1

> At time of writing, starting a lxd-vm instance from a local snapshot takes
> longer than starting a fresh lxd-vm instance and installing ua.

| machine_type  | condition          |
| ------------- | ------------------ |
| lxd-container | num_scenarios > 7  |
| lxd-vm        | never              |
| gcp           | num_scenarios > 5  |
| azure         | num_scenarios > 14 |
| aws           | num_scenarios > 11 |

## Integration testing on EC2
The following tox environments allow for testing focal on EC2:

```
  # To test ubuntu-pro-images
  tox -e behave-awspro-20.04
  # To test Canonical cloud images (non-ubuntu-pro)
  tox -e behave-awsgeneric-20.04
```

To run the test for a different release, just update the release version string. For example,
to run AWS pro xenial tests, you can run:

```
tox -e behave-awspro-16.04
```

In order to run EC2 tests, please set up the [pycloudlib toml
file](https://github.com/canonical/pycloudlib/blob/main/pycloudlib.toml.template) with
the required EC2 credentials.

To specifically run non-ubuntu pro tests using canonical cloud-images an
additional token obtained from https://ubuntu.com/pro needs to be set:
  - UACLIENT_BEHAVE_CONTRACT_TOKEN=<your_token>

* To manually run EC2 integration tests with a specific AMI Id provide the
following environment variable to launch your specific  AMI instead of building
a daily ubuntu-advantage-tools image.
```sh
UACLIENT_BEHAVE_REUSE_IMAGE=your-custom-ami tox -e behave-awspro-20.04
```

## Integration testing on Azure
The following tox environments allow for testing focal on Azure:

```
  # To test ubuntu-pro-images
  tox -e behave-azurepro-20.04
  # To test Canonical cloud images (non-ubuntu-pro)
  tox -e behave-azuregeneric-20.04
```

To run the test for a different release, just update the release version string. For example,
to run Azure pro xenial tests, you can run:

```
tox -e behave-azurepro-16.04
```

In order to run Azure tests, please set up the [pycloudlib toml
file](https://github.com/canonical/pycloudlib/blob/main/pycloudlib.toml.template) with
the required Azure credentials.

To specifically run non-ubuntu pro tests using canonical cloud-images an
additional token obtained from https://ubuntu.com/pro needs to be set:
  - UACLIENT_BEHAVE_CONTRACT_TOKEN=<your_token>

* To manually run Azure integration tests with a specific Image Id provide the
following environment variable to launch your specific Image Id instead of building
a daily ubuntu-advantage-tools image.
```sh
UACLIENT_BEHAVE_REUSE_IMAGE=your-custom-image-id tox -e behave-azurepro-20.04
```

## Integration testing on GCP
The following tox environments allow for testing focal on GCP:

```
  # To test ubuntu-pro-images
  tox -e behave-gcppro-20.04
  # To test Canonical cloud images (non-ubuntu-pro)
  tox -e behave-gcpgeneric-20.04
```

To run the test for a different release, just update the release version string. For example,
to run GCP pro xenial tests, you can run:

```
tox -e behave-gcppro-16.04
```

In order to run GCP tests, please set up the [pycloudlib toml
file](https://github.com/canonical/pycloudlib/blob/main/pycloudlib.toml.template) with
the required GCP credentials.

To specifically run non-ubuntu pro tests using canonical cloud-images an
additional token obtained from https://ubuntu.com/pro needs to be set:
  - UACLIENT_BEHAVE_CONTRACT_TOKEN=<your_token>

* To manually run GCP integration tests with a specific Image Id provide the
following environment variable to launch your specific Image Id instead of building
a daily ubuntu-advantage-tools image.
```sh
UACLIENT_BEHAVE_REUSE_IMAGE=your-custom-image-id tox -e behave-gcppro-20.04
```
