> For the complete documentation index, see [llms.txt](https://theiotlearninginitiative.gitbook.io/edgecomputingsolutions/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://theiotlearninginitiative.gitbook.io/edgecomputingsolutions/introduction/stacks/openstack/testing/akraino/refstack.md).

# RefStack

## Refstack

> RefStack provides the Interop Working Group, a.k.a. DefCore committee, the tools for vendors and other users to run API tests against their clouds to provide the Interop Working Group with a reliable overview of what APIs and capabilities are being used in the marketplace. [RefStack](https://wiki.openstack.org/wiki/RefStack)

* [Tests](https://refstack.openstack.org/api/v1/guidelines/2017.09/tests)

## RefStack :: Installation

```bash
user@workstation:~/$ git clone https://github.com/openstack/refstack-client
user@workstation:~/$ cd refstack-client
user@workstation:~/refstack-client$ ./setup_env
```

```bash
user@workstation:~/refstack-client$ find . -name tempest.conf.sample
./etc/tempest.conf.sample
user@workstation:~/refstack-client$ find . -name accounts.yaml.sample
./etc/accounts.yaml.sample
./.tempest/etc/accounts.yaml.sample
./.tempest/.venv/etc/tempest/accounts.yaml.sample
user@workstation:~/refstack-client$
```

```bash
user@workstation:~/refstack-client$ cp ./etc/accounts.yaml.sample ./etc/accounts.yaml
user@workstation:~/refstack-client$ cp ./etc/tempest.conf.sample ./etc/tempest.conf
user@workstation:~/refstack-client$
```

[File etc/accounts.yaml](https://github.com/openstack/refstack-client/blob/master/etc/accounts.yaml.sample)

```bash
user@workstation:~/refstack-client$ nano ./etc/accounts.yaml
```

[File etc/tempest.conf](https://github.com/openstack/refstack-client/blob/master/etc/tempest.conf.sample)

```bash
user@workstation:~/refstack-client$ nano ./etc/tempest.conf
```

```bash
# This is a minimal example of what your tempest.conf file
# can look like. You will need to supply your own values
# and possibly add additional configurations for your cloud
# You can use this as a starting point for configuring
# tempest for the refstack client.

[Default]
debug = True
log_file = tempest.log
use_stderr = False
use_syslog = False

[auth]
test_accounts_file = /home/user/refstack-client/etc/accounts.yaml
use_dynamic_credentials = False
tempest_roles = Member
admin_username = admin
admin_domain_name = Default
admin_project_name = admin
admin_password = Madawaska1*
#admin_domain_scope=false
#default_credentials_domain_name
#username='refstackone'
#project_name=refstackone
#domain_name=refstackone
#domain_scope=false
#password=Madawaska1*

[compute]
max_microversion = latest
image_ref = 3c8995af-4223-42b6-8872-7ed75bf19ddd
image_ref_alt = 3c8995af-4223-42b6-8872-7ed75bf19ddd
flavor_ref = 1
flavor_ref_alt = 1
fixed_network_name = 012afdaa-92f1-401f-8a54-b3e15d43f82f
build_timeout = 196

[compute-feature-enabled]
resize = true

[identity]
auth_version = v3
uri = http://10.10.10.2:5000/v3
uri_v3 = http://10.10.10.2:5000/v3
admin_domain_scope = True
user_unique_last_password_count = 2
user_lockout_duration = 10
user_lockout_failure_attempts = 2

[identity-feature-enabled]
#initial#api_v2 = false
#initial#api_v3 = true
project_tags = True
security_compliance = True
api_v2_admin = False
api_v2 = False

[network]
public_network_id = 012afdaa-92f1-401f-8a54-b3e15d43f82f
floating_network_name = subnet.nano

[object-storage]
operator_role = SwiftOperator
reseller_admin_role = ResellerAdmin

[oslo_concurrency]
lock_path = /tmp/tempest

[service_available]
cinder = true
neutron = true
glance = true
swift = true
nova = true
# Option "heat" from group "service_available" is deprecated for removal (Heat support will be removed from Tempest).
# heat = false

[validation]
run_validation = true
connect_method = floating
auth_method = keypair
```

```bash
user@workstation:~/refstack-client$ source .venv/bin/activate
(.venv) user@workstation:~/refstack-client$
```

```bash
(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token
```

## RefStack :: Hello World

```bash
(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token
2018-03-28 15:14:56,497 refstack_client:139 DEBUG The .testrepository/next-stream file was not found. Assuming subunit results will be stored in file 0.
2018-03-28 15:14:57,266 refstack_client:421 INFO Starting Tempest test...
2018-03-28 15:14:57.845 10230 INFO tempest [-] Using tempest config file /home/user/refstack-client/etc/tempest.conf
2018-03-28 15:14:57.854 10230 WARNING oslo_config.cfg [-] Option "heat" from group "service_available" is deprecated for removal (Heat support will be removed from Tempest). Its value may be silently ignored in the future.
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmp5daxWM
2018-03-28 15:14:59.991 10240 INFO tempest [-] Using tempest config file /home/user/refstack-client/etc/tempest.conf
2018-03-28 15:15:00.000 10240 WARNING oslo_config.cfg [-] Option "heat" from group "service_available" is deprecated for removal (Heat support will be removed from Tempest). Its value may be silently ignored in the future.
2018-03-28 15:15:00.337 10240 INFO tempest.lib.common.preprov_creds [-] TokensV3Test allocated creds:
{'username': 'admin', 'tenant_name': 'admin'}
2018-03-28 15:15:00.337 10240 WARNING tempest.lib.common.preprov_creds [-] Using credentials {'username': 'admin', 'tenant_name': 'admin'} for v3 API calls. Assuming they are valid in the default domain.
2018-03-28 15:15:01.074 10240 INFO tempest.lib.common.rest_client [req-8946e52d-9930-4136-a338-340c1fc36935 ] Request (TokensV3Test:setUpClass): 201 POST http://10.10.10.2:5000/v3/auth/tokens
2018-03-28 15:15:01.534 10240 INFO tempest.lib.common.rest_client [req-ffcb7403-27ce-425e-8331-824e61a28687 ] Request (TokensV3Test:test_create_token): 201 POST http://10.10.10.2:5000/v3/auth/tokens
{0} tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token [0.460717s] ... ok
2018-03-28 15:15:01.545 10240 INFO tempest.lib.common.preprov_creds [-] TokensV3Test returned allocated creds:
{'username': 'admin', 'tenant_name': 'admin', 'domain_name': 'Default'}

======
Totals
======
Ran: 1 tests in 3.0000 sec.
- Passed: 1
- Skipped: 0
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
Sum of execute time for each test: 0.4607 sec.

==============
Worker Balance
==============
- Worker 0 (1 tests) => 0:00:00.460717
2018-03-28 15:15:01,686 refstack_client:481 INFO Tempest test complete.
2018-03-28 15:15:01,686 refstack_client:482 INFO Subunit results located in: /home/user/refstack-client/.tempest/.testrepository/0
2018-03-28 15:15:01,688 refstack_client:485 INFO Number of passed tests: 1
2018-03-28 15:15:01,688 refstack_client:497 INFO JSON results saved in: /home/user/refstack-client/.tempest/.testrepository/0.json
(.venv) user@workstation:~/refstack-client$
```

```bash
(.venv) user@workstation:~/refstack-client$ rm -rf /tmp/tempest
(.venv) user@workstation:~/refstack-client$ rm -rf /tmp/tempest-lock
```

## RefStack :: Compute :: Nova

> Configuration needed...

```bash
(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.compute.
```

```bash
======
Totals
======
Ran: 202 tests in 317.0000 sec.
 - Passed: 71
 - Skipped: 81
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 50
Sum of execute time for each test: 204.4447 sec.
```

## RefStack :: Identity :: Keystone

> Needs lot of understanding :)

Learning

* [Tempest Configuration Guide](https://docs.openstack.org/tempest/latest/configuration.html)
* [What is a Tenant ion OpenStack](https://www.quora.com/What-is-a-tenant-in-openstack)

These are tempest.api.identity tests from

* [RefStack Tests](https://refstack.openstack.org/api/v1/guidelines/2017.09/tests)
* Create 4 users under Dashboard and include them under etc/account.yaml
* Wish List .. Write a script to create X users based on [this](https://docs.openstack.org/mitaka/install-guide-obs/keystone-users.html) script.

```bash
- username: 'refstackone'
  tenant_name: 'refstackone'
  password: 'refstackone'
  ...
- username: 'refstackfour'
  tenant_name: 'refstackfour'
  password: 'refstackfour'
```

1. Run your first test

```bash
(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token
```

1. Check your results

```bash
(.venv) user@workstation:~/refstack-client$ refstack-client list
```

```bash
(.venv) user@workstation:~/refstack-client$ refstack-client test -c etc/tempest.conf -v --
```

```bash
[:(] tempest.api.identity.v3.TestApiDiscovery.test_api_media_types
The test run didn't actually run any tests

[:(] tempest.api.identity.v3.TestApiDiscovery.test_api_version_resources
The test run didn't actually run any tests

[:(] tempest.api.identity.v3.TestApiDiscovery.test_api_version_statuses
The test run didn't actually run any tests

[:(] tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_media_types
The test run didn't actually run any tests

[:(] tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_version_resources

(.venv) user@workstation:~/refstack-client$ refstack-client test -c etc/tempest.conf -v -- tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_version_resources
2018-04-02 12:05:52,926 refstack_client:421 INFO Starting Tempest test...
2018-04-02 12:05:53.504 6689 INFO tempest [-] Using tempest config file /home/user/refstack-client/etc/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list 
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpb3PiyS
2018-04-02 12:05:55.587 6699 INFO tempest [-] Using tempest config file /home/user/refstack-client/etc/tempest.conf
2018-04-02 12:05:55.920 6699 INFO tempest.lib.common.preprov_creds [-] TestApiDiscovery allocated creds:
{'username': 'tempestone', 'tenant_name': 'tempestone'}
2018-04-02 12:05:55.920 6699 WARNING tempest.lib.common.preprov_creds [-] Using credentials {'username': 'tempestone', 'tenant_name': 'tempestone'} for v3 API calls. Assuming they are valid in the default domain.
2018-04-02 12:05:56.653 6699 INFO tempest.lib.common.rest_client [req-5cc02777-882a-4685-8744-10337997f68a ] Request (TestApiDiscovery:setUpClass): 201 POST http://10.10.10.2:5000/v3/auth/tokens
2018-04-02 12:06:56.715 6699 WARNING urllib3.connectionpool [-] Retrying (Retry(total=9, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f352fd95bd0>, 'Connection to 192.168.204.2 timed out. (connect timeout=60)')': /v3: ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f352fd95bd0>, 'Connection to 192.168.204.2 timed out. (connect timeout=60)')



[] tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_version_statuses
[] tempest.api.identity.v3.test_projects.IdentityV3ProjectsTest.test_list_projects_returns_only_authorized_projects
[] tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token
```

````bash
tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token
``

# RefStack :: Image :: Glance

> Need to check Skipped tests

```sh
(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.image.v2.
````

```bash
Totals
======
Ran: 52 tests in 27.0000 sec.
 - Passed: 48
 - Skipped: 4
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 12.2020 sec.
```

## RefStack :: Network :: Neutron

> Only a set of network tests were done... need to run the rest

```bash
(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.network.test_networks.NetworksTest.
```

```bash
======
Totals
======
Ran: 20 tests in 66.0000 sec.
 - Passed: 19
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 1
Sum of execute time for each test: 56.3611 sec.
```

## RefStack :: Object Storage :: Swift

> Configuration required...

```bash
(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.object_storage.
```

```bash
======
Totals
======
Ran: 24 tests in 3.0000 sec.
 - Passed: 0
 - Skipped: 24
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 0.0000 sec.
```

## RefStack :: Volume :: Cinder

> Configuration required...

```bash
(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.volume.
```

```bash
======
Totals
======
Ran: 61 tests in 18.0000 sec.
 - Passed: 0
 - Skipped: 32
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 29
Sum of execute time for each test: 1.1857 sec.
```