Test Suite — pulsar 1.6.1 documentation (original) (raw)

The TestSuite is a testing framework for both synchronous and asynchronous applications and for running tests in parallel on multiple threads or processes. It is used for testing pulsar but it can be used as a test suite for any other library.

Contents

• Test Suite
  • Integration
    * Setuptools Integration
    * Manual Integration
  • Running Tests
    * Writing a Test Case
    * Loading Tests
    * Test files
    * Test labels
  • Options
    * sequential
    * list labels
    * test timeout
  • Http TestClient
  • Test Plugins
    * Benchmark
    * Profile
  • API
    * Test Suite
    * Test Loader
    * Plugin
    * Test Runner
    * Test Result
    * Test Plugin
    * Http Test Client
    * Populate
  • Utilities
    * sequential
    * ActorTestMixin
    * check server

Integration¶

Pulsar test suite can be used in conjunction with setuptools or stand-alone.

Manual Integration¶

If for some reason you don’t want/can’t use the setuptools integration, you can write a standalone script, for example runtests.py:

from pulsar.apps import TestSuite

if name == 'main': TestSuite(description='Test suite for my library').start()

To run the test suite:

Type:

python runtests.py --help

For a list different options/parameters which can be used when running tests.

Running Tests¶

Writing a Test Case¶

Only subclasses of TestCase are collected by this application. An example test case:

import unittest

class MyTest(unittest.TestCase):

async def test_async_test(self):
    result = await async_function()
    self.assertEqual(result, ...)

def test_simple_test(self):
    self.assertEqual(1, 1)

Note

Test functions are asynchronous, when they are coroutine functions or return a Future, synchronous, when they return anything else.

Loading Tests¶

The loading of test cases is controlled by the modules parameter when initialising the TestSuite:

from pulsar.apps import TestSuite

if name == 'main': TestSuite(modules=('tests', 'examples')).start()

When using pulsar test suite - setuptools integration, test modules are specified in the setup.cfg:

[test] test_modules = tests examples

Test files¶

The TestSuite loads tests via the loaderproperty. By default it recursively loads test files inside themodules directories matching:

• test_<label>.py
• <label>_test.py
• tests.py

Test labels¶

To run a specific label:

python runtests.py

When using the setuptools integration:

python setup.py test -a

To list all labels:

It is also possible to run a single test function within a label:

python setup.py test -a .

Options¶

All standard settings can be applied to the test application. In addition, the following options aretest suite specific:

sequential¶

By default, test functions within a TestCaseare run in asynchronous fashion. This means that several test functions may be executed at once depending on their return values.

By specifying the –sequential command line option, the TestSuite forces test functions from a givenTestCase to be run in a sequential way, one after the other:

python runtests.py --sequential

Alternatively, if you need to specify a TestCase which always runs its test functions in a sequential way, you can use the sequential() decorator:

from pulsar.apps.test import sequential

@sequential class MyTestCase(unittest.TestCase): ...

list labels¶

By passing the -l or –list-labels flag to the command line, the full list of test labels available is displayed:

test timeout¶

When running asynchronous tests, it can be useful to set a cap on how long a test function can wait for results. This is what the–test-timeout command line flag does:

python runtests.py --test-timeout 10

Set the test timeout to 10 seconds. Test timeout is only meaningful for asynchronous test function.

Http TestClient¶

The HttpTestClient can be used to test wsgi middleware without going through socket connections.

To use the client in a test function:

async def test_my_wsgi_test(self): http = HttpTestClient(self, wsgi) response = await http.get('http://bla.com/...')

The host part of the url is irrelevant, it can be anything you like.

Test Plugins¶

A TestPlugin is a way to extend the test suite with additionaloptions and behaviours implemented in the various plugin’s callbacks. There are two basic rules for plugins:

• Test plugin classes should subclass TestPlugin.
• Test plugins may implement any of the methods described in the classPlugin interface.

Pulsar ships with two battery-included plugins:

Benchmark¶

BenchMark is a TestPlugin for benchmarking test functions.

To use the plugin follow these three steps:

• Flag a unittest.TestCase class with the __benchmark__ = Trueclass attribute:
  class MyBenchmark(unittest.TestCase):
  benchmark = True
  def test_mybenchmark_function1(self):
  ...
  def test_mybenchmark_function2(self):
  ...
• Run the test suite with the --benchmark command line option.

Alternatively, you can use the bench command from your setup.py file. Simply add the bench entry in the setup.cfg.

[bench] test_modules = tests/bench

The test class can implement additional methods to fine-tune how the benchmark plugin evaluate the performance and display results:

• When implemented, the startUp method is invoked before each run of a test function.
• The time taken to run a test once can be modified by implementing the getTime method which receives as only argument the time interval taken. By default it returns the same time interval.

class pulsar.apps.test.plugins.bench. BenchMark[source]¶

Benchmarking addon for pulsar test suite.

Profile¶

Profile is a TestPlugin for profiling test cases and generating Html reports. It uses the cProfile module from the standard library.

To use the plugin follow these two steps:

• Included it in the test Suite:
  from pulsar.apps.test import TestSuite
  from pulsar.apps.test.plugins import profile
  def suite():
  TestSuite(..., plugins=(..., profile.Profile()))
• Run the test suite with the --profile command line option.

class pulsar.apps.test.plugins.profile. Profile[source]¶

TestPlugin for profiling test cases.

API¶

Test Suite¶

class pulsar.apps.test. TestSuite(callable=None, load_config=True, **params)[source]¶

An asynchronous test suite which works like a task queue.

Each task is a group of test methods in a python TestCase class.

loader¶

Instance of the TestLoader used for loading test cases

monitor_start(monitor)[source]¶

When the monitor starts load all test classes into the queue

Test Loader¶

class pulsar.apps.test.loader. TestLoader(suite)[source]¶

Classes used by the TestSuite to aggregate tests from a list of paths.

The way it works is simple, you give a root directory and a list of submodules where to look for tests.

tags(include=None, exclude=None)[source]¶

Return a generator of tag information

test_files(include=None, exclude=None)[source]¶

List of tag, modules pairs.

Plugin¶

class pulsar.apps.test.result. Plugin[source]¶

Interface for all classes which are part of the TestRunner.

Most classes used by the test application are plugins, for example the TestRunner itself, the TestResult and the TestPlugin.

result = None¶

An optional result

stream = None¶

handle for writing text on the default output.

Set by the TestRunner at runtime.

configure(cfg)[source]¶

Called once just after construction of a TestRunner

and before any test class is loaded.

This is a chance to configure the Plugin or global variables which may affect the way tests are run. If it returns something other than None (for example an abort message) it will stop the configuration of all subsequent plugins and quit the test.

on_start()[source]¶

Called by the TestSuite once only at startup.

This callback is invoked once all tests are loaded but before the test suite starts running them.

on_end()[source]¶

Called by the TestSuite just before it stops.

loadTestsFromTestCase(testcls)[source]¶

Called when loading tests from the testcls class.

Can be used to modify the number of test functions loaded.

startTestClass(testcls)[source]¶

Called just before a testcls runs its tests.

stopTestClass(testcls)[source]¶

Called just after a testcls has run its tests.

startTest(test)[source]¶

Called just before a test function is executed.

This is run just before _pre_setup method.

stopTest(test)[source]¶

Called just after a test function has finished.

This is run just after the _post_teardown method.

before_test_function_run(test, local)[source]¶

Can be used by plugins to manipulate the testbehaviour in the process domain where the test run.

after_test_function_run(test, local)[source]¶

Executed in the test process domain, after the test has finished.

addSuccess(test)[source]¶

Called when a test function succeed

addFailure(test, err)[source]¶

Called when a test function as a (test) failure

addError(test, err)[source]¶

Called when a test function as an (unexpected) error

Test Runner¶

class pulsar.apps.test.result. TestRunner(plugins, stream, writercls=None, descriptions=True)[source]¶

A Plugin for asynchronously running tests.

loadTestsFromTestCase(test_cls)[source]¶

Load all test functions for the test_cls

before_test_function_run(test)[source]¶

Called just before the test is run

after_test_function_run(test)[source]¶

Called just after the test has finished, in the test process domain.

Test Result¶

class pulsar.apps.test.result. TestResult(descriptions=True)[source]¶

A Plugin for collecting results/failures for test runs.

Each Plugin can access the TestRunner resultobject via the result attribute.

startTest(test)[source]¶

Increase the test counter

addError(test, err)[source]¶

Called when an unexpected error has occurred.

err is a tuple of values as returned by sys.exc_info()

addFailure(test, err)[source]¶

Called when an test failure has occurred.

err is a tuple of values as returned by sys.exc_info()

addSkip(test, reason)[source]¶

Called when a test is skipped.

addExpectedFailure(test, err)[source]¶

Called when an expected failure/error occurred.

addUnexpectedSuccess(test)[source]¶

Called when a test was expected to fail, but succeed.

wasSuccessful()[source]¶

Tells whether or not this result was a success

Test Plugin¶

class pulsar.apps.test.plugins.base. TestPlugin[source]¶

Base class for Plugin which can be added to aTestSuite to extend its functionalities.

If the class attribute name is not specified or its value validate as True, an additional setting is added to the configuration. In addition, a TestPlugin can specify several additionalsettings as class attributes. For example, thebenchmark plugin has an additional setting for controlling the number of repetitions:

class Bench(TestPlugin): repeat = pulsar.Setting(type=int, default=1, validator=pulsar.validate_pos_int, desc="Default number of repetition")

name¶

Class attribute used for adding the default pluginsettingto the configuration container of the test suite application. If the attribute is not set, the class name in lower case is used. If set and validate as not True, no new settingis added to the test suite configuration parameters. For example:

class MyPlugin(TestPlugin): name = None

won’t add the default plugin setting.

desc¶

Class attribute used as the description of thesetting. If name is disabled, this attribute is not relevant.

config¶

A Config container created by the TestPluginmetaclass. It collects the default setting, if available, and any additional settings specified as class attributes.

Http Test Client¶

class pulsar.apps.test.wsgi. HttpTestClient(test=None, wsgi=None, **kwargs)[source]¶

A test client for http requests to a WSGI server handlers.

wsgi¶

The WSGI server handler to test

connection_pool¶

alias of DummyConnectionPool

Populate¶

A useful function for populating random data:

from pulsar.apps.test import populate

data = populate('string', 100)

gives you a list of 100 random strings

pulsar.apps.test.populate. populate(datatype='string', size=10, start=None, end=None, converter=None, choice_from=None, **kwargs)[source]¶

Utility function for populating lists with random data.

Useful when populating database with data for fuzzy testing.

Supported data-types

• string
  For example:
  populate('string', 100, min_len=3, max_len=10)
  create a 100 elements list with random strings with random length between 3 and 10
• date
  For example:
  from datetime import date
  populate('date', 200, start=date(1997,1,1), end=date.today())
  create a 200 elements list with random datetime.date objects between start and end
• integer
  For example:
  populate('integer', 200, start=0, end=1000)
  create a 200 elements list with random integers between startand end
• float
  For example:
  populate('float', 200, start=0, end=10)
  create a 200 elements list with random floats between start andend
• choice (elements of an iterable)
  For example:
  populate('choice', 200, choice_from=['pippo','pluto','blob'])
  create a 200 elements list with random elements from choice_from

Utilities¶

sequential¶

pulsar.apps.test.utils. sequential(cls)[source]¶

Decorator for a TestCase which cause its test functions to run sequentially rather than in an asynchronous fashion.

Typical usage:

import unittest

from pulsar.apps.test import sequential

@sequenatial class MyTests(unittest.TestCase): ...

You can also run test functions sequentially when using thesequential flag in the command line.

ActorTestMixin¶

class pulsar.apps.test.utils. ActorTestMixin[source]¶

A mixin for TestCase.

Useful for classes testing spawning of actors. Make sure this is the first class you derive from, before theTestCase, so that the tearDown method is overwritten.

concurrency¶

The concurrency model used to spawn actors via the spawn()method.

spawn_actor(concurrency=None, **kwargs)[source]¶

Spawn a new actor and perform some tests

check server¶

pulsar.apps.test.utils. check_server(name)[source]¶