# Testing ## Testing * [1](https://wiki.openstack.org/wiki/Testing) * [2](http://superuser.openstack.org/articles/a-guide-to-testing-in-openstack/) ## Sphinx * [1](https://docs.readthedocs.io/en/latest/index.html) ## Rally > OpenStack is, undoubtedly, a really huge ecosystem of cooperative services. Rally is a benchmarking tool that answers the question: "How does OpenStack work at scale?". To make this possible, Rally automates and unifies multi-node OpenStack deployment, cloud verification, benchmarking & profiling. Rally does it in a generic way, making it possible to check whether OpenStack is going to work well on, say, a 1k-servers installation under high load. Thus it can be used as a basic tool for an OpenStack CI/CD system that would continuously improve its SLA, performance and stability. [Some Webpage](http://docs.xrally.xyz/projects/openstack/en/0.5.0/overview.html) * ## Tempest > Tempest Testing Project [Homepage](https://docs.openstack.org/tempest/latest/) ## Other * (openstack) module list ```bash user@workstation:~/$ mkdir ma user@workstation:~/$ cd ma user@workstation:~/ma$ virtualenv titaniumcloud Running virtualenv with interpreter /usr/bin/python2 New python executable in /home/user/ma/titaniumcloud/bin/python2 Also creating executable in /home/user/ma/titaniumcloud/bin/python Installing setuptools, pkg_resources, pip, wheel...done. user@workstation:~/ma$ ``` ```bash user@workstation:~/ma$ source titaniumcloud/bin/activate (titaniumcloud) user@workstation:~/ma$ ``` ```bash (titaniumcloud) user@workstation:~/ma$ git clone https://git.openstack.org/openstack/tempest Cloning into 'tempest'... (titaniumcloud) user@workstation:~/ma$ pip install tempest/ ``` ```bash (titaniumcloud) user@workstation:~/ma$ ls titaniumcloud/etc/tempest/ accounts.yaml.sample logging.conf.sample whitelist.yaml (titaniumcloud) user@workstation:~/ma$ export TEMPEST_CONFIG_DIR=titaniumcloud/etc/tempest/ (titaniumcloud) user@workstation:~/ma$ ``` * ```bash (titaniumcloud) user@workstation:~/ma$ tempest account-generator -c titaniumcloud/etc/tempest/tempest.conf --os-username admin --os-password Madawaska1* --os-project-name admin --os-tenant-name admin --os-domain-name Default -r 1 --with-admin -i 3 accounts_file.yaml ``` ```bash (titaniumcloud) user@workstation:~/ma$ easy_install tox (titaniumcloud) user@workstation:~/ma$ cd tempest/ (titaniumcloud) user@workstation:~/ma/tempest$ tox -epy27 --notest (titaniumcloud) user@workstation:~/ma/tempest$ source .tox/all-plugin/bin/activate ``` ## Old ```bash (.venv) user@workstation:~/$ git clone https://git.openstack.org/openstack/tempest (.venv) user@workstation:~/$ cd tempest (.venv) user@workstation:~/tempest$ cd tempest (.venv) user@workstation:~/tempest$ pip install . ``` ``` (.venv) user@workstation:~/tempest$ mkdir titaniumcloud (.venv) user@workstation:~/tempest$ cd titaniumcloud/ ``` ```bash (.venv) user@workstation:~/tempest/titaniumcloud$ tempest init 2018-03-28 10:21:42.910 16597 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf (.venv) user@workstation:~/tempest/titaniumcloud$ ``` ```bash (.venv) user@workstation:~/tempest/titaniumcloud$ ls etc logs tempest_lock (.venv) user@workstation:~/tempest/titaniumcloud$ ``` ```bash (.venv) user@workstation:~/tempest/titaniumcloud$ tempest account-generator -h ``` ```bash (.venv) user@workstation:~/tempest/titaniumcloud$ tempest account-generator -c /home/user/tempest/titaniumcloud/etc/tempest.conf --os-username admin --os-password Madawaska1* --os-project-name admin --os-tenant-name admin --os-domain-name Default -r 1 --with-admin -i 3 accounts_file.yaml ``` ```bash (.venv) user@workstation:~/tempest/titaniumcloud$ sudo apt install python-testrepository ``` ```bash (.venv) user@workstation:~/tempest/titaniumcloud$ tempest verify-config -o etc/tempest.conf ``` ## Other * ```bash user@workstation:~$ sudo apt install python-pip user@workstation:~$ virtualenv titaniumcloud user@workstation:~$ source titaniumcloud/bin/activate (titaniumcloud) user@workstation:~$ ``` ```bash (titaniumcloud) user@workstation:~$ pip install tempest/ ``` ````bash (titaniumcloud) user@workstation:~$ export TEMPEST_CONFIG_DIR=titaniumcloud/etc/tempest/ `` # Commands ```sh (.venv) user@workstation:~/tempest$ tempest workspace list 2018-03-28 11:06:55.278 21000 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf +---------------+----------------------------------+ | Name | Path | +---------------+----------------------------------+ | cloud-01 | /home/user/cloud-01 | | titaniumcloud | /home/user/tempest/titaniumcloud | +---------------+----------------------------------+ (.venv) user@workstation:~/tempest$ ```` ```bash (.venv) user@workstation:~/tempest$ tempest list-plugins +------+------------+ | Name | EntryPoint | +------+------------+ +------+------------+ ``` ## Refstack > OpenStack Interoperability. RefStack is a source of tools for OpenStack interoperability testing. [Homepage](https://refstack.openstack.org/#/) > > *Refstack Design* RefStack provides the Interop Working Group, a.k.a. DefCore committee, the tools for vendors and other users to run API tests against their clouds to provide the Interop Working Group with a reliable overview of what APIs and capabilities are being used in the marketplace. [Wiki](https://wiki.openstack.org/wiki/RefStack) ```bash user@workstation:~/$ git clone https://github.com/openstack/refstack-client user@workstation:~/$ cd refstack-client user@workstation:~/refstack-client$ ./setup_env ``` ```bash user@workstation:~/refstack-client$ find . -name tempest.conf.sample ./etc/tempest.conf.sample user@workstation:~/refstack-client$ find . -name accounts.yaml.sample ./etc/accounts.yaml.sample ./.tempest/etc/accounts.yaml.sample ./.tempest/.venv/etc/tempest/accounts.yaml.sample user@workstation:~/refstack-client$ ``` ```bash user@workstation:~/refstack-client$ cp ./etc/accounts.yaml.sample ./etc/accounts.yaml user@workstation:~/refstack-client$ cp ./etc/tempest.conf.sample ./etc/tempest.conf user@workstation:~/refstack-client$ ``` ```bash user@workstation:~/refstack-client$ nano ./etc/accounts.yaml ``` ```bash user@workstation:~/refstack-client$ nano ./etc/tempest.conf ``` ```bash user@workstation:~/refstack-client$ source .venv/bin/activate (.venv) user@workstation:~/refstack-client$ ``` ```bash (.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token ``` ```bash (.venv) user@workstation:~/refstack-client$ refstack-client test -c ./etc/tempest.conf -v -- --regex tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token 2018-03-28 15:14:56,497 refstack_client:139 DEBUG The .testrepository/next-stream file was not found. Assuming subunit results will be stored in file 0. 2018-03-28 15:14:57,266 refstack_client:421 INFO Starting Tempest test... 2018-03-28 15:14:57.845 10230 INFO tempest [-] Using tempest config file /home/user/refstack-client/etc/tempest.conf 2018-03-28 15:14:57.854 10230 WARNING oslo_config.cfg [-] Option "heat" from group "service_available" is deprecated for removal (Heat support will be removed from Tempest). Its value may be silently ignored in the future. running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmp5daxWM 2018-03-28 15:14:59.991 10240 INFO tempest [-] Using tempest config file /home/user/refstack-client/etc/tempest.conf 2018-03-28 15:15:00.000 10240 WARNING oslo_config.cfg [-] Option "heat" from group "service_available" is deprecated for removal (Heat support will be removed from Tempest). Its value may be silently ignored in the future. 2018-03-28 15:15:00.337 10240 INFO tempest.lib.common.preprov_creds [-] TokensV3Test allocated creds: {'username': 'admin', 'tenant_name': 'admin'} 2018-03-28 15:15:00.337 10240 WARNING tempest.lib.common.preprov_creds [-] Using credentials {'username': 'admin', 'tenant_name': 'admin'} for v3 API calls. Assuming they are valid in the default domain. 2018-03-28 15:15:01.074 10240 INFO tempest.lib.common.rest_client [req-8946e52d-9930-4136-a338-340c1fc36935 ] Request (TokensV3Test:setUpClass): 201 POST http://10.10.10.2:5000/v3/auth/tokens 2018-03-28 15:15:01.534 10240 INFO tempest.lib.common.rest_client [req-ffcb7403-27ce-425e-8331-824e61a28687 ] Request (TokensV3Test:test_create_token): 201 POST http://10.10.10.2:5000/v3/auth/tokens {0} tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token [0.460717s] ... ok 2018-03-28 15:15:01.545 10240 INFO tempest.lib.common.preprov_creds [-] TokensV3Test returned allocated creds: {'username': 'admin', 'tenant_name': 'admin', 'domain_name': 'Default'} ====== Totals ====== Ran: 1 tests in 3.0000 sec. - Passed: 1 - Skipped: 0 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute time for each test: 0.4607 sec. ============== Worker Balance ============== - Worker 0 (1 tests) => 0:00:00.460717 2018-03-28 15:15:01,686 refstack_client:481 INFO Tempest test complete. 2018-03-28 15:15:01,686 refstack_client:482 INFO Subunit results located in: /home/user/refstack-client/.tempest/.testrepository/0 2018-03-28 15:15:01,688 refstack_client:485 INFO Number of passed tests: 1 2018-03-28 15:15:01,688 refstack_client:497 INFO JSON results saved in: /home/user/refstack-client/.tempest/.testrepository/0.json (.venv) user@workstation:~/refstack-client$ ``` ### REfStack :: Identity * [How to configure and run Tempest for testing OpenStack Keystone Identity V3 API?](https://www.symantec.com/connect/blogs/how-configure-and-run-tempest-testing-openstack-keystone-identity-v3-api)