Testing
Frameworks

Basic idea for the new cw-2.py's timeout function to have an extra argument that prevents running the function right away.

pass
Testing

This tries to address the problem observed in this Kata which tries to wrap the user function in the tests.

def replicate(times, num):
    if times <= 0: return []
    return [num] + replicate(times-1, num)
Testing
Frameworks

More testing on expect_error. By the nature of try..except statement, it is possible to specify multiple exception types to expect.

Testing
Frameworks

This Kumite is a showcase for further possible improvements of Python testing framework cw-2.py, which has been going through major changes. You can see already implemented features here (on Preview site).

Proposal

Improved expect_error(), idea by Blind4Basics

As-is:

  • Test if the function throws an error. It doesn't matter which exception it exactly throws.

To-be:

  • Test if the function throws an error, which is an instance of the given Exception class.
# BaseException >> Exception
def f1(): raise Exception()

# BaseException >> Exception >> ArithmeticError >> ZeroDivisionError
def f2(): return 1 // 0

# BaseException >> Exception >> LookupError >> KeyError
def f3(): return {}[1]

def f0(): pass
Testing
Frameworks

This kumite illustrates some of the new features of the Python test framework (a.k.a. cw-2.py) through a simple kata format.

Example Task

Compute the sum of two integers.

Constraints

-10**9 <= a, b <= 10**9

Example Solutions

  1. add: Correct solution
  2. add1: Incorrect solution
  3. add2: Solution that timeouts
  4. add3: Solution that cheats
  5. add4: Solution that throws error (see how the error is formatted; files and lines are shown properly now)

You can uncomment each line at the bottom of Code section to see how the tests combat the invalid solutions in different ways.

Extra Things in the code section

  1. You can use and print Unicode characters without error!

Example Test Fixture

  1. @Test.describe(name)
  2. @Test.it(name)
  3. @Test.timeout(sec)
  4. Test.assert_equals and Test.assert_not_equals are non-blocking by default
  5. Test.assert_not_equals can now prevent cheating with magic methods

For more complete list of features and guide on how to write a test fixture, consult this page on GitHub Wiki.

Code
Diff
  • # Correct solution
    def add(a, b): return a + b
    
    # Incorrect solution
    def add1(a, b): return a | b
    
    # Solution that timeouts
    def add2(a, b):
        while a: a, b = a-1, b+1
        return b
    
    # Solution that cheats
    def add3(a, b):
        class X(object):
            def __eq__(self, other): return True
            def __ne__(self, other): return True
            def __gt__(self, other): return True
            def __lt__(self, other): return True
            def __ge__(self, other): return True
            def __le__(self, other): return True
        return X()
    
    # Solution that throws error
    def add4(a, b):
        if a > 10: raise NotImplementedError()
        return a + b
        
    #add = add1
    #add = add2
    #add = add3
    #add = add4
    
    # Other things to demonstrate
    print('가나다')
  • 1616
            def __ne__(self, other): return True
    
    1717
            def __gt__(self, other): return True
    
    1818
            def __lt__(self, other): return True
    
    1919
            def __ge__(self, other): return True
    
    2020
            def __le__(self, other): return True
    
    2121
        return X()
    
    22+
    23+
    # Solution that throws error
    
    24+
    def add4(a, b):
    
    25+
        if a > 10: raise NotImplementedError()
    
    26+
        return a + b
    
    2222
        
    
    2323
    #add = add1
    
    2424
    #add = add2
    
    2525
    #add = add3
    
    31+
    #add = add4
    
    32+
    33+
    # Other things to demonstrate
    
    34+
    print('가나다')
    
Testing
Frameworks

Please go to same kumite on the preview site to run these tests correctly :)

Example Task

Compute the sum of two integers.

Constraints

-10**9 <= a, b <= 10**9

Example Solutions

  1. add: Correct solution
  2. add1: Incorrect solution
  3. add2: Solution that timeouts
  4. add3: Solution that cheats

You can uncomment each line at the bottom of Code section to see how the tests combat the invalid solutions in different ways.

Example Test Fixture

  1. @Test.describe(name)
  2. @Test.it(name)
  3. @Test.timeout(sec)
  4. Test.assert_equals and Test.assert_not_equals are non-blocking by default
  5. Test.assert_not_equals can now prevent cheating with magic methods

For more complete list of features and guide on how to write a test fixture, consult this page on GitHub Wiki.

# Correct solution
def add(a, b): return a + b

# Incorrect solution
def add1(a, b): return a | b

# Solution that timeouts
def add2(a, b):
    while a: a, b = a-1, b+1
    return b

# Solution that cheats
def add3(a, b):
    class X(object):
        def __eq__(self, other): return True
        def __ne__(self, other): return True
        def __gt__(self, other): return True
        def __lt__(self, other): return True
        def __ge__(self, other): return True
        def __le__(self, other): return True
    return X()
    
#add = add1
#add = add2
#add = add3
Testing

Outline

This is a proposal for CodeWars test framework for Python to improve it in many ways (and be more consistent with other languages' frameworks).

Unlike my previous Kumite, this one is designed to completely replace cw-2.py in the runner repo.

Changes / Improvements

Individual Testing / Logging Functions

  • Make testing functions non-blocking
  • Change the expression inside assert_not_equals so that it can prevent operator injection hacks
  • Provide a way to log and test Unicode strings without Unicode-related error
  • Provide a utility for timeout
  • Provide pass, fail and assert_approx_equals

Describe / It

  • Build the decorator version of describe and it, so the text fixture may look like this (and the decorator itself runs the code, so it doesn't need a separate runner function):
@describe('describe-text')
def describe1():
    @it('it-text', before=f1, after=f2)
    def it1():
        # some test function
    @it('it-text')
    def it2():
        # some test function
  • Properly close describe and it blocks in the test output
  • Print the running times of describe and it blocks
  • Provide before and after for describe and it

Changelog

v1.1

  • Replace the timeout utility with decorator version.

v1.2

  • Make the whole code (including fixture examples) compatible with Python 2 using six.
Code
Diff
  • from __future__ import print_function
    import re, six
    range = six.moves.range
    
    
    class AssertException(Exception):
        pass
    
    
    '''Fix the dreaded Unicode Error Trap'''
    def uni_print(*args, **kwargs):
        from sys import stdout
        sep = kwargs.get('sep', ' ')
        end = kwargs.get('end', '\n')
        file = kwargs.get('file', stdout)
        
        def _replace(c):
            if ord(c) >= 128: return u'&#{};'.format(ord(c))
            return c
        def _escape(s):
            escaped = ''.join(_replace(c) for c in six.text_type(s))
            escaped = re.sub(r'\\u([\da-f]{4})', lambda m: '&#{};'.format(int(m.group(1), 16)), escaped)
            escaped = re.sub(r'\\U([\da-f]{8})', lambda m: '&#{};'.format(int(m.group(1), 16)), escaped)
            return escaped
            
        six.print_(*map(_escape, args), sep=_escape(sep), end=_escape(end), file=file)
    
    
    def format_message(message):
        def _replace(c):
            if ord(c) >= 65536: return r'\U' + hex(ord(c))[2:].zfill(8)
            if ord(c) >= 128: return r'\u' + hex(ord(c))[2:].zfill(4)
            return c
        def _escape(s): return ''.join(_replace(c) for c in s)
        return _escape(message.replace("\n", "<:LF:>"))
    
    
    def display(type, message, label="", mode=""):
        print("\n<{0}:{1}:{2}>{3}".format(type.upper(), mode.upper(), label, format_message(message)))
    
    
    def expect(passed=None, message=None, allow_raise=False):
        if passed:
            display('PASSED', 'Test Passed')
        else:
            message = message or "Value is not what was expected"
            display('FAILED', message)
            if allow_raise:
                raise AssertException(message)
    
    
    def assert_equals(actual, expected, message=None, allow_raise=False):
        equals_msg = "{0} should equal {1}".format(repr(actual), repr(expected))
        if message is None:
            message = equals_msg
        else:
            message += ": " + equals_msg
    
        expect(actual == expected, message, allow_raise)
    
    
    def assert_not_equals(actual, expected, message=None, allow_raise=False):
        equals_msg = "{0} should not equal {1}".format(repr(actual), repr(expected))
        if message is None:
            message = equals_msg
        else:
            message += ": " + equals_msg
    
        expect(not (actual == expected), message, allow_raise)
    
    
    def expect_error(message, function):
        passed = False
        try:
            function()
        except:
            passed = True
        expect(passed, message)
    
    
    def pass_(): expect(True)
    def fail(message): expect(False, message)
    
    
    def assert_approx_equals(actual, expected, margin=1e-9, message=None, allow_raise=False):
        equals_msg = "{0} should be close to {1} with absolute or relative margin of {2}".format(
            repr(actual), repr(expected), repr(margin))
        if message is None: message = equals_msg
        else: message += ": " + equals_msg
        div = max(abs(actual), abs(expected), 1)
        expect(abs((actual - expected) / div) < margin, message, allow_raise)
    
    
    '''
    Usage:
    @describe('describe text')
    def describe1():
        @it('it text')
        def it1():
            # some test cases...
    '''
    def _timed_block_factory(opening_text):
        from timeit import default_timer as timer
        from traceback import format_exception
        from sys import exc_info
        
        def _timed_block_decorator(s, before=None, after=None):
            display(opening_text, s)
            def wrapper(func):
                if callable(before): before()
                time = timer()
                try: func()
                except:
                    fail('Unexpected exception raised')
                    tb_str = ''.join(format_exception(*exc_info()))
                    display('ERROR', tb_str)
                display('COMPLETEDIN', '{:.2f}'.format((timer() - time) * 1000))
                if callable(after): after()
            return wrapper
        return _timed_block_decorator
    
    describe = _timed_block_factory('DESCRIBE')
    it = _timed_block_factory('IT')
    
    
    '''
    Timeout utility
    Usage:
    @timeout(sec)
    def some_tests():
        any code block...
    Note: Timeout value can be a float.
    '''
    def timeout(sec):
        def wrapper(func):
            from multiprocessing import Process
            process = Process(target=func)
            process.start()
            process.join(sec)
            if process.is_alive():
                fail('Exceeded time limit of {:.3f} seconds'.format(sec))
                process.terminate()
                process.join()
        return wrapper
    
    
    '''Old-style Fixture'''
    describe('Old-style Describe')
    it('Old-style It')
    assert_equals(0, 0)
    assert_equals(0, 1)
    print('<COMPLETEDIN::>')
    it('Old-style It 2')
    assert_equals('a', 'a')
    assert_equals('a', 'b')
    print('<COMPLETEDIN::>')
    print('<COMPLETEDIN::>')
    
    
    '''Sample Fixture #1'''
    @describe('Sample Fixture #1')
    def sample_describe_1():
        @it('Sample Testcase #1-1')
        def sample_it_1():
            assert_equals(0, 0)
            assert_equals(0, 1)
            assert_not_equals(0, 2)
            pass_()
            fail('This should fail')
        @it('Sample Testcase #1-2')
        def sample_it_2():
            expect_error('ZeroDivisionError', lambda: 0 / 0)
            assert_equals(0, 0 / 0)
            assert_equals(1, 1, 'This is not run due to exception')
        @it('Sample Testcase #1-3')
        def sample_it_3():
            assert_equals('abc', 'abc')
            # source code doesn't support utf-8 chars, but you can at least log and test unicode
            assert_equals(u'\uac00 \ub098 \ub2e4', u'\uac00 \ub098 \ub2e4')
            uni_print(1, 'a', u'\uac00 \ub098 \ub2e4', [2, 'b', u'\uac00'])
            assert_equals(u'\uac00 \ub098 \ub2e4', 'a b c')
    
    
    '''Sample Fixture #2: Featuring Before and After'''
    @describe('Sample Fixture #2')
    def sample_describe_2():
        a = {0}
        def before():
            a.add(len(a))
        @it('Sample Testcase #2-1', before=before, after=before)
        def sample_it_1():
            assert_equals(a, {0, 1})
        @it('Sample Testcase #2-2')
        def sample_it_2():
            assert_equals(a, {0, 1, 2})
    
    
    '''Sample Fixture #3: Featuring Timeout'''
    @describe('Sample Fixture #3')
    def sample_describe_3():
        @it('Sample Testcase #3-1')
        def sample_it_1():
            @timeout(0.01)
            def count():
                for _ in range(100): pass
                pass_()
        @it('Sample Testcase #3-2')
        def sample_it_2():
            @timeout(0.01)
            def count():
                for _ in range(10**10): pass
                pass_()
    
    
    '''Sample Fixture #4: Featuring assert_approx_equals'''
    @describe('Sample Fixture #4')
    def sample_describe_4():
        @it('Sample Testcase #4-1')
        def sample_it_1():
            assert_approx_equals(1, 1 + 1e-10, 1e-9)
            assert_approx_equals(1, 1 + 1e-7, 1e-9)
            assert_approx_equals(-1, -1 - 1e-10, 1e-9)
            assert_approx_equals(-1, -1 - 1e-7, 1e-9)
        @it('Sample Testcase #4-2')
        def sample_it_2():
            assert_approx_equals(0, 1e-10, 1e-9)
            assert_approx_equals(0, 1e-7, 1e-9)
            assert_approx_equals(0, -1e-10, 1e-9)
            assert_approx_equals(0, -1e-7, 1e-9)
    
  • 1
    #from __future__ import print_function
    
    1+
    from __future__ import print_function
    
    2+
    import re, six
    
    3+
    range = six.moves.range
    
    4+
    22
    33
    class AssertException(Exception):
    
    44
        pass
    
    55
    66
    77
    '''Fix the dreaded Unicode Error Trap'''
    
    8
    _print = print
    
    9
    def print(*args, sep=' ', end='\n'):
    
    10
        from io import StringIO
    
    11
        def _escape(s): return s.encode('ascii', 'xmlcharrefreplace').decode('ascii')
    
    12
        sio = StringIO()
    
    13
        _print(*args, sep=sep, end=end, file=sio)
    
    14
        _print(_escape(sio.getvalue()))
    
    15
        sio.close()
    
    11+
    def uni_print(*args, **kwargs):
    
    12+
        from sys import stdout
    
    13+
        sep = kwargs.get('sep', ' ')
    
    14+
        end = kwargs.get('end', '\n')
    
    15+
        file = kwargs.get('file', stdout)
    
    16+
        
    
    17+
        def _replace(c):
    
    18+
            if ord(c) >= 128: return u'&#{};'.format(ord(c))
    
    19+
            return c
    
    20+
        def _escape(s):
    
    21+
            escaped = ''.join(_replace(c) for c in six.text_type(s))
    
    22+
            escaped = re.sub(r'\\u([\da-f]{4})', lambda m: '&#{};'.format(int(m.group(1), 16)), escaped)
    
    23+
            escaped = re.sub(r'\\U([\da-f]{8})', lambda m: '&#{};'.format(int(m.group(1), 16)), escaped)
    
    24+
            return escaped
    
    25+
            
    
    26+
        six.print_(*map(_escape, args), sep=_escape(sep), end=_escape(end), file=file)
    
    1616
    1717
    1818
    def format_message(message):
    
    19
        return message.replace("\n", "<:LF:>")
    
    30+
        def _replace(c):
    
    31+
            if ord(c) >= 65536: return r'\U' + hex(ord(c))[2:].zfill(8)
    
    32+
            if ord(c) >= 128: return r'\u' + hex(ord(c))[2:].zfill(4)
    
    33+
            return c
    
    2020
    2121
    2222
    def expect(passed=None, message=None, allow_raise=False):
    
    23
        if passed: print("\n<PASSED::>Test Passed")
    
    43+
        if passed:
    
    44+
            display('PASSED', 'Test Passed')
    
    2424
        else:
    
    2525
            message = message or "Value is not what was expected"
    
    26
            print("\n<FAILED::>{0}".format(message))
    
    27
            if allow_raise: raise AssertException(message)
    
    47+
            display('FAILED', message)
    
    48+
            if allow_raise:
    
    49+
                raise AssertException(message)
    
    2828
    2929
    30
    '''Fix the blocking asserts to non-blocking'''
    
    3131
    def assert_equals(actual, expected, message=None, allow_raise=False):
    
    3232
        equals_msg = "{0} should equal {1}".format(repr(actual), repr(expected))
    
    33
        if message is None: message = equals_msg
    
    42
    '''
    
    4343
    def assert_not_equals(actual, expected, message=None, allow_raise=False):
    
    4444
        equals_msg = "{0} should not equal {1}".format(repr(actual), repr(expected))
    
    45
        if message is None: message = equals_msg
    
    46
        else: message += ": " + equals_msg
    
    4747
        expect(not (actual == expected), message, allow_raise)
    
    4848
    4949
    5050
    def expect_error(message, function):
    
    5151
        passed = False
    
    52
        try: function()
    
    53
        except: passed = True
    
    74+
        try:
    
    75+
            function()
    
    76+
        except:
    
    77+
            passed = True
    
    5959
    def fail(message): expect(False, message)
    
    6969
    7070
    71
    def display(type, message, label="", mode=""):
    
    72
        print("\n<{0}:{1}:{2}>{3}".format(type.upper(), mode.upper(), label, format_message(message)))
    
    73
    74
    7575
    '''
    
    76
    Modern-Style Describe & It
    
    7777
    Usage:
    
    7878
    @describe('describe text')
    
    7979
    def describe1():
    
    8585
        from timeit import default_timer as timer
    
    8686
        from traceback import format_exception
    
    8787
        from sys import exc_info
    
    8888
        
    
    8989
        def _timed_block_decorator(s, before=None, after=None):
    
    90
            print('<{}::>{}'.format(opening_text, s))
    
    108+
            display(opening_text, s)
    
    9191
            def wrapper(func):
    
    9292
                if callable(before): before()
    
    9393
                time = timer()
    
    9494
                try: func()
    
    9595
                except:
    
    9696
                    fail('Unexpected exception raised')
    
    97
                    tb_str = ''.join(format_exception(*exc_info())).replace('\n', '<:LF:>')
    
    98
                    print('<ERROR::>' + tb_str)
    
    99
                print('<COMPLETEDIN::>{}'.format(round((timer() - time) * 1000, 2)))
    
    115+
                    tb_str = ''.join(format_exception(*exc_info()))
    
    116+
                    display('ERROR', tb_str)
    
    117+
                display('COMPLETEDIN', '{:.2f}'.format((timer() - time) * 1000))
    
    100100
                if callable(after): after()
    
    101101
            return wrapper
    
    102102
        return _timed_block_decorator
    
    103103
    104104
    describe = _timed_block_factory('DESCRIBE')
    
    105105
    it = _timed_block_factory('IT')
    
    106106
    107107
    108108
    '''
    
    109109
    Timeout utility
    
    110110
    Usage:
    
    111
    with run_with_timeout(func, tuple_of_args, timeout_in_seconds) as run:
    
    112
        Test.assert_equals(run.get(), expected_value)
    
    129+
    @timeout(sec)
    
    130+
    def some_tests():
    
    131+
        any code block...
    
    113113
    Note: Timeout value can be a float.
    
    114114
    '''
    
    115
    class run_with_timeout(object):
    
    116
        def __init__(self, func, inputs, sec):
    
    117
            from multiprocessing import Process, Queue
    
    118
            def timeout_wrapper(func, inputs, q):
    
    124
        def __enter__(self):
    
    125
            self.p.start()
    
    126
            return self
    
    127
        def get(self):
    
    128
            if self.result is None: self.result = self.q.get(timeout=self.sec)
    
    129
            return self.result
    
    130
        def __exit__(self, typ, val, traceback):
    
    131
            self.q.close()
    
    132
            self.p.terminate()
    
    133
            self.p.join()
    
    134
            if traceback: fail('Exceeded time limit of {:.3f} seconds'.format(self.sec))
    
    135
            return True
    
    134+
    def timeout(sec):
    
    135+
        def wrapper(func):
    
    136+
            from multiprocessing import Process
    
    137+
            process = Process(target=func)
    
    138+
            process.start()
    
    139+
            process.join(sec)
    
    140+
            if process.is_alive():
    
    141+
                fail('Exceeded time limit of {:.3f} seconds'.format(sec))
    
    142+
                process.terminate()
    
    143+
                process.join()
    
    144+
        return wrapper
    
    136136
    137137
    138138
    '''Old-style Fixture'''
    
    139139
    describe('Old-style Describe')
    
    140140
    it('Old-style It')
    
    141141
    assert_equals(0, 0)
    
    142142
    assert_equals(0, 1)
    
    143143
    print('<COMPLETEDIN::>')
    
    144144
    it('Old-style It 2')
    
    145145
    assert_equals('a', 'a')
    
    146146
    assert_equals('a', 'b')
    
    147147
    print('<COMPLETEDIN::>')
    
    148148
    print('<COMPLETEDIN::>')
    
    149149
    150150
    165165
            assert_equals(1, 1, 'This is not run due to exception')
    
    166166
        @it('Sample Testcase #1-3')
    
    167167
        def sample_it_3():
    
    168168
            assert_equals('abc', 'abc')
    
    169169
            # source code doesn't support utf-8 chars, but you can at least log and test unicode
    
    170
            assert_equals('\uac00 \ub098 \ub2e4', '\uac00 \ub098 \ub2e4')
    
    171
            print('\uac00 \ub098 \ub2e4')
    
    172
            assert_equals('\uac00 \ub098 \ub2e4', 'a b c')
    
    179+
            assert_equals(u'\uac00 \ub098 \ub2e4', u'\uac00 \ub098 \ub2e4')
    
    180+
            uni_print(1, 'a', u'\uac00 \ub098 \ub2e4', [2, 'b', u'\uac00'])
    
    181+
            assert_equals(u'\uac00 \ub098 \ub2e4', 'a b c')
    
    173173
    174174
    175175
    '''Sample Fixture #2: Featuring Before and After'''
    
    176176
    @describe('Sample Fixture #2')
    
    177177
    def sample_describe_2():
    
    178178
        a = {0}
    
    179179
        def before():
    
    180180
            a.add(len(a))
    
    181181
        @it('Sample Testcase #2-1', before=before, after=before)
    
    182182
        def sample_it_1():
    
    183183
            assert_equals(a, {0, 1})
    
    184184
        @it('Sample Testcase #2-2')
    
    185185
        def sample_it_2():
    
    186186
            assert_equals(a, {0, 1, 2})
    
    187187
    188188
    189189
    '''Sample Fixture #3: Featuring Timeout'''
    
    190190
    @describe('Sample Fixture #3')
    
    191191
    def sample_describe_3():
    
    192
        def wait_count(n):
    
    193
            for _ in range(n): pass
    
    194
            return n
    
    195195
        @it('Sample Testcase #3-1')
    
    196196
        def sample_it_1():
    
    197
            with run_with_timeout(wait_count, (100,), 0.01) as run:
    
    198
                assert_equals(run.get(), 100)
    
    203+
            @timeout(0.01)
    
    204+
            def count():
    
    205+
                for _ in range(100): pass
    
    206+
                pass_()
    
    199199
        @it('Sample Testcase #3-2')
    
    200200
        def sample_it_2():
    
    201
            with run_with_timeout(wait_count, (10 ** 10,), 0.01) as run:
    
    202
                assert_equals(run.get(), 10 ** 10)
    
    209+
            @timeout(0.01)
    
    210+
            def count():
    
    211+
                for _ in range(10**10): pass
    
    212+
                pass_()
    
    203203
    204204
    205205
    '''Sample Fixture #4: Featuring assert_approx_equals'''
    
    206206
    @describe('Sample Fixture #4')
    
    207207
    def sample_describe_4():
    
    208208
        @it('Sample Testcase #4-1')
    
    209209
        def sample_it_1():
    
    210210
            assert_approx_equals(1, 1 + 1e-10, 1e-9)
    
    211211
            assert_approx_equals(1, 1 + 1e-7, 1e-9)
    
    212212
            assert_approx_equals(-1, -1 - 1e-10, 1e-9)
    
    213213
            assert_approx_equals(-1, -1 - 1e-7, 1e-9)
    
    214214
        @it('Sample Testcase #4-2')
    
    215215
        def sample_it_2():
    
    216216
            assert_approx_equals(0, 1e-10, 1e-9)
    
    217217
            assert_approx_equals(0, 1e-7, 1e-9)
    
    218218
            assert_approx_equals(0, -1e-10, 1e-9)
    
    219219
            assert_approx_equals(0, -1e-7, 1e-9)
    
Testing

Outline

This is a proposal for CodeWars test framework for Python to improve it in many ways (and be more consistent with other languages' frameworks).

Unlike my previous Kumite, this one is designed to completely replace cw-2.py in the runner repo.

Changes / Improvements

Individual Testing / Logging Functions

  • Make testing functions non-blocking
  • Change the expression inside assert_not_equals so that it can prevent operator injection hacks
  • Provide a way to log and test Unicode strings without Unicode-related error
  • Provide a utility for timeout
  • Provide pass, fail and assert_approx_equals

Describe / It

  • Build the decorator version of describe and it, so the text fixture may look like this (and the decorator itself runs the code, so it doesn't need a separate runner function):
@describe('describe-text')
def describe1():
    @it('it-text', before=f1, after=f2)
    def it1():
        # some test function
    @it('it-text')
    def it2():
        # some test function
  • Properly close describe and it blocks in the test output
  • Print the running times of describe and it blocks
  • Provide before and after for describe and it
#from __future__ import print_function

class AssertException(Exception):
    pass


'''Fix the dreaded Unicode Error Trap'''
_print = print
def print(*args, sep=' ', end='\n'):
    from io import StringIO
    def _escape(s): return s.encode('ascii', 'xmlcharrefreplace').decode('ascii')
    sio = StringIO()
    _print(*args, sep=sep, end=end, file=sio)
    _print(_escape(sio.getvalue()))
    sio.close()


def format_message(message):
    return message.replace("\n", "<:LF:>")


def expect(passed=None, message=None, allow_raise=False):
    if passed: print("\n<PASSED::>Test Passed")
    else:
        message = message or "Value is not what was expected"
        print("\n<FAILED::>{0}".format(message))
        if allow_raise: raise AssertException(message)


'''Fix the blocking asserts to non-blocking'''
def assert_equals(actual, expected, message=None, allow_raise=False):
    equals_msg = "{0} should equal {1}".format(repr(actual), repr(expected))
    if message is None: message = equals_msg
    else: message += ": " + equals_msg
    expect(actual == expected, message, allow_raise)


'''
Fix the blocking asserts to non-blocking
Also change the expected formula from `actual != expected` to `not (actual == expected)`
so that using this assertion can prevent the `==` / `!=` injection hack
'''
def assert_not_equals(actual, expected, message=None, allow_raise=False):
    equals_msg = "{0} should not equal {1}".format(repr(actual), repr(expected))
    if message is None: message = equals_msg
    else: message += ": " + equals_msg
    expect(not (actual == expected), message, allow_raise)


def expect_error(message, function):
    passed = False
    try: function()
    except: passed = True
    expect(passed, message)


'''Additional test functions: pass, fail, and assert_approx_equals'''
def pass_(): expect(True)
def fail(message): expect(False, message)


def assert_approx_equals(actual, expected, margin=1e-9, message=None, allow_raise=False):
    equals_msg = "{0} should be close to {1} with absolute or relative margin of {2}".format(
        repr(actual), repr(expected), repr(margin))
    if message is None: message = equals_msg
    else: message += ": " + equals_msg
    div = max(abs(actual), abs(expected), 1)
    expect(abs((actual - expected) / div) < margin, message, allow_raise)


def display(type, message, label="", mode=""):
    print("\n<{0}:{1}:{2}>{3}".format(type.upper(), mode.upper(), label, format_message(message)))


'''
Modern-Style Describe & It
Usage:
@describe('describe text')
def describe1():
    @it('it text')
    def it1():
        # some test cases...
'''
def _timed_block_factory(opening_text):
    from timeit import default_timer as timer
    from traceback import format_exception
    from sys import exc_info
    
    def _timed_block_decorator(s, before=None, after=None):
        print('<{}::>{}'.format(opening_text, s))
        def wrapper(func):
            if callable(before): before()
            time = timer()
            try: func()
            except:
                fail('Unexpected exception raised')
                tb_str = ''.join(format_exception(*exc_info())).replace('\n', '<:LF:>')
                print('<ERROR::>' + tb_str)
            print('<COMPLETEDIN::>{}'.format(round((timer() - time) * 1000, 2)))
            if callable(after): after()
        return wrapper
    return _timed_block_decorator

describe = _timed_block_factory('DESCRIBE')
it = _timed_block_factory('IT')


'''
Timeout utility
Usage:
with run_with_timeout(func, tuple_of_args, timeout_in_seconds) as run:
    Test.assert_equals(run.get(), expected_value)
Note: Timeout value can be a float.
'''
class run_with_timeout(object):
    def __init__(self, func, inputs, sec):
        from multiprocessing import Process, Queue
        def timeout_wrapper(func, inputs, q):
            q.put(func(*inputs))
        self.sec = sec
        self.q = Queue()
        self.p = Process(target=timeout_wrapper, args=(func, inputs, self.q))
        self.result = None
    def __enter__(self):
        self.p.start()
        return self
    def get(self):
        if self.result is None: self.result = self.q.get(timeout=self.sec)
        return self.result
    def __exit__(self, typ, val, traceback):
        self.q.close()
        self.p.terminate()
        self.p.join()
        if traceback: fail('Exceeded time limit of {:.3f} seconds'.format(self.sec))
        return True


'''Old-style Fixture'''
describe('Old-style Describe')
it('Old-style It')
assert_equals(0, 0)
assert_equals(0, 1)
print('<COMPLETEDIN::>')
it('Old-style It 2')
assert_equals('a', 'a')
assert_equals('a', 'b')
print('<COMPLETEDIN::>')
print('<COMPLETEDIN::>')


'''Sample Fixture #1'''
@describe('Sample Fixture #1')
def sample_describe_1():
    @it('Sample Testcase #1-1')
    def sample_it_1():
        assert_equals(0, 0)
        assert_equals(0, 1)
        assert_not_equals(0, 2)
        pass_()
        fail('This should fail')
    @it('Sample Testcase #1-2')
    def sample_it_2():
        expect_error('ZeroDivisionError', lambda: 0 / 0)
        assert_equals(0, 0 / 0)
        assert_equals(1, 1, 'This is not run due to exception')
    @it('Sample Testcase #1-3')
    def sample_it_3():
        assert_equals('abc', 'abc')
        # source code doesn't support utf-8 chars, but you can at least log and test unicode
        assert_equals('\uac00 \ub098 \ub2e4', '\uac00 \ub098 \ub2e4')
        print('\uac00 \ub098 \ub2e4')
        assert_equals('\uac00 \ub098 \ub2e4', 'a b c')


'''Sample Fixture #2: Featuring Before and After'''
@describe('Sample Fixture #2')
def sample_describe_2():
    a = {0}
    def before():
        a.add(len(a))
    @it('Sample Testcase #2-1', before=before, after=before)
    def sample_it_1():
        assert_equals(a, {0, 1})
    @it('Sample Testcase #2-2')
    def sample_it_2():
        assert_equals(a, {0, 1, 2})


'''Sample Fixture #3: Featuring Timeout'''
@describe('Sample Fixture #3')
def sample_describe_3():
    def wait_count(n):
        for _ in range(n): pass
        return n
    @it('Sample Testcase #3-1')
    def sample_it_1():
        with run_with_timeout(wait_count, (100,), 0.01) as run:
            assert_equals(run.get(), 100)
    @it('Sample Testcase #3-2')
    def sample_it_2():
        with run_with_timeout(wait_count, (10 ** 10,), 0.01) as run:
            assert_equals(run.get(), 10 ** 10)


'''Sample Fixture #4: Featuring assert_approx_equals'''
@describe('Sample Fixture #4')
def sample_describe_4():
    @it('Sample Testcase #4-1')
    def sample_it_1():
        assert_approx_equals(1, 1 + 1e-10, 1e-9)
        assert_approx_equals(1, 1 + 1e-7, 1e-9)
        assert_approx_equals(-1, -1 - 1e-10, 1e-9)
        assert_approx_equals(-1, -1 - 1e-7, 1e-9)
    @it('Sample Testcase #4-2')
    def sample_it_2():
        assert_approx_equals(0, 1e-10, 1e-9)
        assert_approx_equals(0, 1e-7, 1e-9)
        assert_approx_equals(0, -1e-10, 1e-9)
        assert_approx_equals(0, -1e-7, 1e-9)
Testing

This shows a piece of test code in Python that is improved over the default cw-2 (and consistent with other languages' test frameworks) in the following aspects:

Initial release (v1.0)

  • Utilize the with-block to support proper indentation of describe and it blocks in the code
  • Properly close describe and it blocks in the test output
  • Print the running times of describe and it blocks
  • Make testing functions non-blocking

v1.1

  • Provide a way to log Unicode strings for users (does not work for test framework)

v1.2

  • Provide before and after for describe and it
  • Provide a utility for timeout
  • Version 1: Function version
  • Version 2: Context manager version (looks cleaner)

Run the tests and see how the test output looks like.

Testing

This shows a piece of test code in Python that is improved over the default cw-2 (and consistent with other languages' test frameworks) in the following aspects:

Initial release (v1.0)

  • Utilize the with-block to support proper indentation of describe and it blocks in the code
  • Properly close describe and it blocks in the test output
  • Print the running times of describe and it blocks
  • Make testing functions non-blocking

v1.1

  • Provide a way to log Unicode strings for users (does not work for test framework)

Run the tests and see how the test output looks like.

def fibonacci(n):
    if n <= 1: return 1
    if n > 35: raise RuntimeError('Too large input')
    return fibonacci(n-1) + fibonacci(n-2)