Testing Framework

Testing Framework is an object-oriented approach to Programmer Tests pioneered in Kent Beck's 1994 Testing Framework package for Smalltalk, later renamed SUnit (Smalltalk Unit). Beck's approach and code have since been adopted and ported to numerous other environments and languages.

Have an object (Test Case) manage the execution of a single test. Each test should be isolated from the others, so it creates all its data before executing and destroys it when it's done (setUp() and tearDown()). Composites of tests (Test Suite) allow many tests to be run together. Passing a Collecting Parameter (Test Collector/Test Result) to the tests as they run allows the collection of statistics. That's the framework - three objects (Test Case, Test Suite, Test Result).

There are simpler ways of running automated tests, and many, many more complicated ways, but this architecture seems to hit a sweet spot.


Originated in

www.xprogramming.com Simple Smalltalk Testing: With Patterns Kent Beck (reprint of the October 1994 *Smalltalk Report* article)

Chapter 30 "Simple Smalltalk Testing", in Kent Beck's Guide to Better Smalltalk by Kent Beck, Donald G. Firesmith (Editor), Cambridge University Press, December 1998, ISBN 9780521644372


Source code Regression Testing libraries, by language:

(see also a similar list at xprogramming.com )

Ada Unit - Ada

Ant Unit = task library for the Apache Ant build tool.

As Unit - (ASUnit) for Action Script [in progress]

As Tuce - (ASTUce) for Action Script 3.0

ACT - Advanced CUint Test for C on LINUX, Windows, Embedded system See kldp.net

Cee Unit - "cUnit" for C. A Unit testing framework for C that depends on glib.

cfix - xUnit testing framework for Win32. Minimizes effort to author tests. Tests are compiled into a DLL rather than an EXE. A testrunner is provided to (selectively) run fixtures of one or more DLLs. GPL licensed. See cfix.sourceforge.net

CGreen [Cee Green] - Open source Unit testing framework for C. Includes process separation of tests and mock functions. See sourceforge.net

Check Framework - A mature well-tested unit test framework for C

CUT - C Unit Tester system. See www.falvotech.com for version 2.4, and www.falvotech.com for CUT 3.0pre1. Portable across all platforms which support Python; does not rely on any libraries (except its own), and is not an SUnit-clone (it relies on a Test Collector to aggregate tests). Suitable for embedded unit testing. CUT 3 promises to be easier to use from within IDEs via error messages resembling the error reports from popular compilers.

CUnit [Cu Nit] - Standard 'C', Platform independent unit testing framework - cunit.sourceforge.net

Cu Test - C. Works on Windows with MSVC and BCC32 (does not depend on glibc)

In The Guard - A simple C test harness intended for testing C libraries during installation.

MinUnit - A minimal unit testing framework for C - www.jera.com

Cutter [Cutter Testing Framework] - An easy to use unit testing framework for C - cutter.sourceforge.net

CUnitWin32 [CeeUnitWin32] - Lean and mean framework with truly independent tests (e.g. global/static variables are initialised between tests). See code.google.com

C2Unit - Yet another C unit testing framework - github.com

cfix - xUnit testing framework for Win32. Minimizes effort to author tests. Tests are compiled into a DLL rather than an EXE. A testrunner is provided to (selectively) run fixtures of one or more DLLs. GPL licensed. See cfix.sourceforge.net

Cpp Test Harness (a modernized and upgraded flavor of Cpp Unit Lite)

Cpp Unit does C# as well as C++; 1.12 version shows some weird runtime errors

Cpp Utx Overview - an excellent alternative to Cpp Unit, though you need to implement parts yourself.

Cutee, C++ Unit Testing Easy Environment. See codesink.org

Managed C++ - Cs Unit (available for free under GPL)

Test Soon - C++ with Macro Metaprogramming. See testsoon.sf.net

TUT - C++ Templates. See tut.dozen.ru

Any Unit - addin for Microsoft Visual Studio.NET to support any C++ unit testing framework through customizable layouts. See www.anyunit.com

Unit Test++ (this should be first in the list :-) --RS) - See www.gamesfromwithin.com

CUnitWin32 [CeeUnitWin32] - Lean and mean framework with truly independent tests (e.g. global/static variables are initialised between tests). See code.google.com

Cute Test - with eclipse plug-in, modern header-only design

Catch Test - modern header-only design

Cs Unit (available for free under GPL)

And see also the list of test frameworks for .NET, listed below.

See the list of Common Lisp test frameworks at www.cliki.net

See the misnamed Common Lisp Unit which describes the CL Regression Test framework RT by Richard Waters

CLOS-unit a CLOS implementation for CLOS/Lisp unit testing. See a.die.supsi.ch

Emacs Lisp (Regress El), Lisp Unit, Common Lisp Unit -- (Not "Star Unit" libraries, but useful, nonetheless.)

cfcUnit - A unit testing framework for Cold Fusion Components. See www.cfcunit.org

MXUnit - Cold Fusion. Unit Test Framework and Eclipse Plug-in for CFMX 6-7-8. See mxunit.org

Rocket Unit - Cold Fusion. Unit Testing Framework. Single CFC implementation. See rocketunit.riaforge.org

Bubelen Project - Bubelen is a Delphi integrated system for building and maintaining bubbles. A bubble is a piece of code for testing, developing, profiling, debugging, and stress testing. Bubelen is open source.

Delphi Unit - A couple of different Delphi testing frameworks

Eiffel Unit - Unit test framework for Eiffel (formally known as ETest or Eiffel Test)

Ga Unit - for Gauche

Gro Unit - for Groovy

Quick Check - an automated "agitator"-style test generator

Haskell Unit - xUnit-style tester, for Test Driven Development

Icon Unit - Icon Language unit test suite

Concordi On - acceptance testing from plain English descriptions of behaviour. See www.concordion.org

Lobo, Continuous Tuning - for performance testing and monitoring. See it at www.oncast.com.br .

Test En Gee (TestNG) - Advanced testing framework for Java. See www.testng.org

Java Unit - "JUnit" library for Java. (a Wiki-unfriendly name). There's also Jakarta Cactus: a server-side unit testing framework that extends JUnit.

actiWATE - Java, for testing web applications, emulates browser, supports Java Script. See www.actiwate.com

Db Unit = JUnit extension targeted for database-driven projects.

Jameleon - An Automated Testing Tool - jameleon.sourceforge.net (in Java, for general acceptance testing)

Jweb Unit = Facade Pattern refactoring on top of Http Unit and Java Unit.

Qe Test - for Java, hosted in Xalan-J

Tag Unit - for JSP custom tags

WebUI for JUnit - JUnit UI framework allows running your unit tests through web browser. www.softxp.com

Html Unit - based on the apache httpclient utilities, for use within JUnit

J2MEUnit - for Java 2 Micro Edition - j2meunit.sourceforge.net

a good list of Java Language testing tools - opensourcetesting.org

Jut Land ("Java Unit Testing - Light, Adaptable 'n' Discreet") by Kevlin Henney -- [not downloadable?]

Ie Unit - Java Script testing framework for Internet Explorer

As Tuce - (ASTUce) for Java Script, JScript, Action Script, etc.

Test Simple - TAP-based testing framework

Kay Unit - for K (Kay Language) and KSQL

lunit - A unit testing framework for Lua - www.mroth.net

Matlab (Matlab Language)

The MATLAB Unit Test Framework - www.mathworks.com

MUnit - xtargets.com

mlUnit_2008a - www.mathworks.co.uk

Unit testing tools - www.mathworks.com

Unit Kit - Strictly speaking, of Mac OS X only, as it is tightly integrated with XCode.

Perl Unit - for Perl - a fairly straight port of Java Unit to Perl

Perl Tap - the basis of the standard Perl testing framework.

Lectro Test - automatic, specification-based testing for Perl.

Perl Actor - automated executable acceptance test framework for Perl.

Perl Test Class - another xUnit framework for Perl, built on top of Perl Tap

Php Unit - for PHP3

Simple Test - PHP unit tester and Web Tester with Mock Object generation

Pl Unit - unit testing framework for PL/SQL Oracle 8i and higher

Pl Sql Unit - (aka utPLSQL) for PL/SQL (Oracle 7.3.4 and higher)

Doc Test - for Python: test cases serve as documentation examples and vice versa.

Python Unit - for Python Language (various offerings)

Test Oob - extensions for Python Unit

Py Test - the framework used for Py Py

Pikzie [Pikzie Testing Framework] - An easy to write/debug Unit Testing Framework for Python. - pikzie.sourceforge.net

RSpec - for Ruby Language - rspec.info

Scheme Unit - as minimalistic as Scheme itself

Sh Unit - for Bourne-derived shells

Smalltalk Unit - the original; for the Smalltalk Language

Sml Unit (SMLUnit is included in the SML# distributions)

SQL-PL

db2unit - Unit testing framework for IBM DB2 for LUW: github.com

Deja Gnu - The GNU testing framework, based on Expect (which in turn is built on Tcl).

Xo Unit - for Extended Object Tcl (Extended Object Tcl)

TransactSQL:

Tsql Unit - for Transact-SQL (Microsoft SQL Server)

Com Unit - for Microsoft Visual Basic and COM

Cs Unit - Visual Basic .NET (available for free under GPL)

ExcelUnit - see Vba Unit

Simply Vb Unit - a VB6 unit testing framework

Vba Unit - for Visual Basic For Applications (which gets around Vb Unit's incompatibility with VBA)

Vb Lite Unit - a really light-weight test platform for VB and VBA that works pretty darn well.

Vb Unit - Visual Basic (for VB6) [See Also Dot Net Unit and "Dot Net platform" below]

See the list of test frameworks for .NET, listed below.

Fox Unit - an Open Source unit testing framework for VFP. See www.foxunit.org

Xb Unit - for Xbase++

XML Schema Unit Test (SUT): sut.sourceforge.net

Xml Unit - for assertions about XML using Junit and Java JAXP

Xslt Unit - for XSLT Stylesheets

Xo Unit - for Extended Object Tcl (Extended Object Tcl) of the Tool Command Language (Tcl)

Multi-Language Testing Frameworks:

Tet Works - (aka TET) Open Group multi-lingual test framework

For the Dot Net platform (a grouping)

Cs Unit - for the Dot Net platform (Cs Unit is available for free under the GPL)

Dot Net Unit - for the Dot Net platform

Dot Unit - for the Dot Net platform

Harness It - .Net languages (commercial tool with "a modest fee")

Mb Unit - Unit testing and Model-Based testing. See mbunit.tigris.org

NUnitAsp - Extension to NUnit for testing ASP.NET applications:
nunitasp.sourceforge.net

Nunit Framework - NUnit for the .Net platform. See sourceforge.net

Ex Unity - "X-Unity" testing for .Net platform (name Wikized), commercial tool

Power Shell Scripts For Testing - to write tests in Power Shell, which can access, run and test functionality in the Dot Net CLR.

Zane Bug - Advanced unit testing tool for .Net platform (NUnit compatibility).

"Open Source Testing Tools in C#", a listing:
csharp-source.net

For HTTP:

Http Unit - for web server testing

Html Unit - for web server testing

Web Inject - Web/HTTP Test Framework - www.webinject.org

Selenium - client-side web application testing - selenium.thoughtworks.com

Diesel Test - in Delphi 5, for Windows NT

Tellurium - Automated Web Testing Framework - code.google.com

See Zope Unit for list of Testing Frameworks that run in the Zope Application Server.

Works in all languages:

Ne Unit - a GUI front end for multiple back end Testing Frameworks. Currently C/C++ & Python.

Data Driven:

Concordi On - acceptance testing from plain English descriptions of behaviour. See www.concordion.org

Fit Wiki / FIT / Framework For Integrated Test = multi-language table driven acceptance testing tool

Fit Nesse = FIT and tools, all rolled into one

Forte Tool Unit - Forte 4GL (TOOL)

Jx Unit = XML driven testing (for Java)

Automated Test Generation:

JTest by Parasoft (Java Test)

Agitar's Agitator (Agitars Agitator)

NTest by Incenteus

Model-Based Tesing (MBT):

Test Optimal, xml based scripts, Data-Driven Testing(DDT), requirements tracking, see TestOptimal.com

"lint" Type "Smell Detectors":

Diesel Test - scripted web testing in Delphi 5, for Windows NT

The Grinder - "The Grinder, a Java Load Testing Framework"

Open Sta (Open System Testing Architecture) - A distributed software testing architecture based on CORBA.

RUBiS - "RUBiS can be used from a web browser for testing purposes or with the provided benchmarking tool." - rubis.objectweb.org


See also:

xprogramming.com - Ron Jeffries's list. (If the URL fails, use Url Mangling.)

www.manageability.org - "Open Source Automated Test Tools Written in Java"

www.opensourcetesting.org - "Java unit testing tools"

java-source.net - "Open Source Testing Tools in Java"

java-source.net - "Open Source Web Testing Tools in Java"

csharp-source.net - "Open Source Testing Tools in C#"

en.wikipedia.org - Wikipedia list, much like this page


How do I do test-then-code with C in embedded systems? Is there anywhere I can go where people are discussing this? I'm getting the idea that it's possible - even on simple PIC or AVR processors. Has anyone tried it?


You've cobbled up an equivalent under Bourne shells with functions. Contact me for details. One trick I found useful has been to put unit tests (for shell files to be sourced) after a "am I being sourced" guard, and run the unit tests if being executed. - Joseph Beckenbaumer. -- KFrede


I need an Email Unit ... pop, smtp, imap, ... David Escala


As some of our test suites get large, we need to speed up our tests so we can run them quickly. Toplink In Memory allows us to stub out the database so that we can continue to run all tests before checking in! -- Ted Ogrady


One of the pieces of information that is demonstrably missing about xUnit lore is when the first version of sUnit came into existence. It can't be any later than 1995, I'd expect, since that's when C3 happened, and thus, XP came into being. I'm pretty sure sUnit existed at least at that time. But, did it exist even before then? If so, when? -- Samuel Falvo


I would appreciate it if folks (or that Shark thingie :) ) would stop removing my updates to CUT, a unit testing package that I wrote. Thank you. :) -- Samuel Falvo


See original on c2.com