RefStack

Refstack

RefStack provides the Interop Working Group, a.k.a. DefCore committee, the tools for vendors and other users to run API tests against their clouds to provide the Interop Working Group with a reliable overview of what APIs and capabilities are being used in the marketplace. RefStack

Tests

RefStack :: Installation

user@workstation:~/$ git clone https://github.com/openstack/refstack-client
user@workstation:~/$ cd refstack-client
user@workstation:~/refstack-client$ ./setup_env

user@workstation:~/refstack-client$ find . -name tempest.conf.sample
./etc/tempest.conf.sample
user@workstation:~/refstack-client$ find . -name accounts.yaml.sample
./etc/accounts.yaml.sample
./.tempest/etc/accounts.yaml.sample
./.tempest/.venv/etc/tempest/accounts.yaml.sample
user@workstation:~/refstack-client$

user@workstation:~/refstack-client$ cp ./etc/accounts.yaml.sample ./etc/accounts.yaml
user@workstation:~/refstack-client$ cp ./etc/tempest.conf.sample ./etc/tempest.conf
user@workstation:~/refstack-client$

File etc/accounts.yaml

user@workstation:~/refstack-client$ nano ./etc/accounts.yaml

File etc/tempest.conf

user@workstation:~/refstack-client$ nano ./etc/tempest.conf

# This is a minimal example of what your tempest.conf file
# can look like. You will need to supply your own values
# and possibly add additional configurations for your cloud
# You can use this as a starting point for configuring
# tempest for the refstack client.

[Default]
debug = True
log_file = tempest.log
use_stderr = False
use_syslog = False

[auth]
test_accounts_file = /home/user/refstack-client/etc/accounts.yaml
use_dynamic_credentials = False
tempest_roles = Member
admin_username = admin
admin_domain_name = Default
admin_project_name = admin
admin_password = Madawaska1*
#admin_domain_scope=false
#default_credentials_domain_name
#username='refstackone'
#project_name=refstackone
#domain_name=refstackone
#domain_scope=false
#password=Madawaska1*

[compute]
max_microversion = latest
image_ref = 3c8995af-4223-42b6-8872-7ed75bf19ddd
image_ref_alt = 3c8995af-4223-42b6-8872-7ed75bf19ddd
flavor_ref = 1
flavor_ref_alt = 1
fixed_network_name = 012afdaa-92f1-401f-8a54-b3e15d43f82f
build_timeout = 196

[compute-feature-enabled]
resize = true

[identity]
auth_version = v3
uri = http://10.10.10.2:5000/v3
uri_v3 = http://10.10.10.2:5000/v3
admin_domain_scope = True
user_unique_last_password_count = 2
user_lockout_duration = 10
user_lockout_failure_attempts = 2

[identity-feature-enabled]
#initial#api_v2 = false
#initial#api_v3 = true
project_tags = True
security_compliance = True
api_v2_admin = False
api_v2 = False

[network]
public_network_id = 012afdaa-92f1-401f-8a54-b3e15d43f82f
floating_network_name = subnet.nano

[object-storage]
operator_role = SwiftOperator
reseller_admin_role = ResellerAdmin

[oslo_concurrency]
lock_path = /tmp/tempest

[service_available]
cinder = true
neutron = true
glance = true
swift = true
nova = true
# Option "heat" from group "service_available" is deprecated for removal (Heat support will be removed from Tempest).
# heat = false

[validation]
run_validation = true
connect_method = floating
auth_method = keypair

user@workstation:~/refstack-client$ source .venv/bin/activate
(.venv) user@workstation:~/refstack-client$

(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token

RefStack :: Hello World

(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token
2018-03-28 15:14:56,497 refstack_client:139 DEBUG The .testrepository/next-stream file was not found. Assuming subunit results will be stored in file 0.
2018-03-28 15:14:57,266 refstack_client:421 INFO Starting Tempest test...
2018-03-28 15:14:57.845 10230 INFO tempest [-] Using tempest config file /home/user/refstack-client/etc/tempest.conf
2018-03-28 15:14:57.854 10230 WARNING oslo_config.cfg [-] Option "heat" from group "service_available" is deprecated for removal (Heat support will be removed from Tempest). Its value may be silently ignored in the future.
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmp5daxWM
2018-03-28 15:14:59.991 10240 INFO tempest [-] Using tempest config file /home/user/refstack-client/etc/tempest.conf
2018-03-28 15:15:00.000 10240 WARNING oslo_config.cfg [-] Option "heat" from group "service_available" is deprecated for removal (Heat support will be removed from Tempest). Its value may be silently ignored in the future.
2018-03-28 15:15:00.337 10240 INFO tempest.lib.common.preprov_creds [-] TokensV3Test allocated creds:
{'username': 'admin', 'tenant_name': 'admin'}
2018-03-28 15:15:00.337 10240 WARNING tempest.lib.common.preprov_creds [-] Using credentials {'username': 'admin', 'tenant_name': 'admin'} for v3 API calls. Assuming they are valid in the default domain.
2018-03-28 15:15:01.074 10240 INFO tempest.lib.common.rest_client [req-8946e52d-9930-4136-a338-340c1fc36935 ] Request (TokensV3Test:setUpClass): 201 POST http://10.10.10.2:5000/v3/auth/tokens
2018-03-28 15:15:01.534 10240 INFO tempest.lib.common.rest_client [req-ffcb7403-27ce-425e-8331-824e61a28687 ] Request (TokensV3Test:test_create_token): 201 POST http://10.10.10.2:5000/v3/auth/tokens
{0} tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token [0.460717s] ... ok
2018-03-28 15:15:01.545 10240 INFO tempest.lib.common.preprov_creds [-] TokensV3Test returned allocated creds:
{'username': 'admin', 'tenant_name': 'admin', 'domain_name': 'Default'}

======
Totals
======
Ran: 1 tests in 3.0000 sec.
- Passed: 1
- Skipped: 0
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
Sum of execute time for each test: 0.4607 sec.

==============
Worker Balance
==============
- Worker 0 (1 tests) => 0:00:00.460717
2018-03-28 15:15:01,686 refstack_client:481 INFO Tempest test complete.
2018-03-28 15:15:01,686 refstack_client:482 INFO Subunit results located in: /home/user/refstack-client/.tempest/.testrepository/0
2018-03-28 15:15:01,688 refstack_client:485 INFO Number of passed tests: 1
2018-03-28 15:15:01,688 refstack_client:497 INFO JSON results saved in: /home/user/refstack-client/.tempest/.testrepository/0.json
(.venv) user@workstation:~/refstack-client$

(.venv) user@workstation:~/refstack-client$ rm -rf /tmp/tempest
(.venv) user@workstation:~/refstack-client$ rm -rf /tmp/tempest-lock

RefStack :: Compute :: Nova

Configuration needed...

(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.compute.

======
Totals
======
Ran: 202 tests in 317.0000 sec.
 - Passed: 71
 - Skipped: 81
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 50
Sum of execute time for each test: 204.4447 sec.

RefStack :: Identity :: Keystone

Needs lot of understanding :)

Learning

These are tempest.api.identity tests from

RefStack Tests
Create 4 users under Dashboard and include them under etc/account.yaml
Wish List .. Write a script to create X users based on this script.

- username: 'refstackone'
  tenant_name: 'refstackone'
  password: 'refstackone'
  ...
- username: 'refstackfour'
  tenant_name: 'refstackfour'
  password: 'refstackfour'

Run your first test

(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token

Check your results

(.venv) user@workstation:~/refstack-client$ refstack-client list

(.venv) user@workstation:~/refstack-client$ refstack-client test -c etc/tempest.conf -v --

[:(] tempest.api.identity.v3.TestApiDiscovery.test_api_media_types
The test run didn't actually run any tests

[:(] tempest.api.identity.v3.TestApiDiscovery.test_api_version_resources
The test run didn't actually run any tests

[:(] tempest.api.identity.v3.TestApiDiscovery.test_api_version_statuses
The test run didn't actually run any tests

[:(] tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_media_types
The test run didn't actually run any tests

[:(] tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_version_resources

(.venv) user@workstation:~/refstack-client$ refstack-client test -c etc/tempest.conf -v -- tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_version_resources
2018-04-02 12:05:52,926 refstack_client:421 INFO Starting Tempest test...
2018-04-02 12:05:53.504 6689 INFO tempest [-] Using tempest config file /home/user/refstack-client/etc/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list 
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpb3PiyS
2018-04-02 12:05:55.587 6699 INFO tempest [-] Using tempest config file /home/user/refstack-client/etc/tempest.conf
2018-04-02 12:05:55.920 6699 INFO tempest.lib.common.preprov_creds [-] TestApiDiscovery allocated creds:
{'username': 'tempestone', 'tenant_name': 'tempestone'}
2018-04-02 12:05:55.920 6699 WARNING tempest.lib.common.preprov_creds [-] Using credentials {'username': 'tempestone', 'tenant_name': 'tempestone'} for v3 API calls. Assuming they are valid in the default domain.
2018-04-02 12:05:56.653 6699 INFO tempest.lib.common.rest_client [req-5cc02777-882a-4685-8744-10337997f68a ] Request (TestApiDiscovery:setUpClass): 201 POST http://10.10.10.2:5000/v3/auth/tokens
2018-04-02 12:06:56.715 6699 WARNING urllib3.connectionpool [-] Retrying (Retry(total=9, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f352fd95bd0>, 'Connection to 192.168.204.2 timed out. (connect timeout=60)')': /v3: ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f352fd95bd0>, 'Connection to 192.168.204.2 timed out. (connect timeout=60)')



[] tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_version_statuses
[] tempest.api.identity.v3.test_projects.IdentityV3ProjectsTest.test_list_projects_returns_only_authorized_projects
[] tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token

tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token
``

# RefStack :: Image :: Glance

> Need to check Skipped tests

```sh
(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.image.v2.

Totals
======
Ran: 52 tests in 27.0000 sec.
 - Passed: 48
 - Skipped: 4
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 12.2020 sec.

RefStack :: Network :: Neutron

Only a set of network tests were done... need to run the rest

(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.network.test_networks.NetworksTest.

======
Totals
======
Ran: 20 tests in 66.0000 sec.
 - Passed: 19
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 1
Sum of execute time for each test: 56.3611 sec.

RefStack :: Object Storage :: Swift

Configuration required...

(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.object_storage.

======
Totals
======
Ran: 24 tests in 3.0000 sec.
 - Passed: 0
 - Skipped: 24
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 0.0000 sec.

RefStack :: Volume :: Cinder

Configuration required...

(.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.volume.

======
Totals
======
Ran: 61 tests in 18.0000 sec.
 - Passed: 0
 - Skipped: 32
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 29
Sum of execute time for each test: 1.1857 sec.

PreviousAkraino NextTempest

Last updated 6 years ago