Package org.testng

Class TestNG

java.lang.Object
org.testng.TestNG
public class TestNG extends Object
This class is the main entry point for running tests in the TestNG framework. Users can create their own TestNG object and invoke it in many different ways:
  • On an existing testng.xml
  • On a synthetic testng.xml, created entirely from Java
  • By directly setting the test classes
You can also define which groups to include or exclude, assign parameters, etc...

The command line parameters are:

  • -d outputdir: specify the output directory
  • -testclass class_name: specifies one or several class names
  • -testjar jar_name: specifies the jar containing the tests
  • -sourcedir src1;src2: ; separated list of source directories (used only when javadoc annotations are used)
  • -target
  • -groups
  • -testrunfactory
  • -listener

Please consult documentation for more details.

FIXME: should support more than simple paths for suite xmls

See Also:

  • Field Details

    • LOGGER

      private static final Logger LOGGER
      This class' log4testng Logger.

    • DEFAULT_COMMAND_LINE_SUITE_NAME

      public static final String DEFAULT_COMMAND_LINE_SUITE_NAME
      The default name for a suite launched from the command line
      See Also:

    • DEFAULT_COMMAND_LINE_TEST_NAME

      public static final String DEFAULT_COMMAND_LINE_TEST_NAME
      The default name for a test launched from the command line
      See Also:

    • DEFAULT_THREADPOOL_FACTORY

      private static final String DEFAULT_THREADPOOL_FACTORY
      See Also:

    • DEFAULT_OUTPUTDIR

      public static final String DEFAULT_OUTPUTDIR
      The default name of the result's output directory (keep public, used by Eclipse).
      See Also:

    • m_instance

      private static TestNG m_instance

    • m_jCommander

      private static com.beust.jcommander.JCommander m_jCommander

    • m_commandLineMethods

      private List<String> m_commandLineMethods

    • m_suites

      protected List<XmlSuite> m_suites

    • m_cmdlineSuites

      private List<XmlSuite> m_cmdlineSuites

    • m_outputDir

      private String m_outputDir

    • m_includedGroups

      private String[] m_includedGroups

    • m_excludedGroups

      private String[] m_excludedGroups

    • m_isJUnit

      private Boolean m_isJUnit

    • m_isMixed

      private Boolean m_isMixed

    • m_useDefaultListeners

      protected boolean m_useDefaultListeners

    • m_failIfAllTestsSkipped

      private boolean m_failIfAllTestsSkipped

    • m_listenersToSkipFromBeingWiredIn

      private final List<String> m_listenersToSkipFromBeingWiredIn

    • m_testRunnerFactory

      private ITestRunnerFactory m_testRunnerFactory

    • m_classListeners

      private final Map<Class<? extends IClassListener>,IClassListener> m_classListeners

    • m_testListeners

      private final Map<Class<? extends ITestListener>,ITestListener> m_testListeners

    • m_suiteListeners

      private final Map<Class<? extends ISuiteListener>,ISuiteListener> m_suiteListeners

    • m_reporters

      private final Map<Class<? extends IReporter>,IReporter> m_reporters

    • m_dataProviderListeners

      private final Map<Class<? extends IDataProviderListener>,IDataProviderListener> m_dataProviderListeners

    • m_dataProviderInterceptors

      private final Map<Class<? extends IDataProviderInterceptor>,IDataProviderInterceptor> m_dataProviderInterceptors

    • m_executorFactory

      private IExecutorFactory m_executorFactory

    • DEFAULT_VERBOSE

      public static final Integer DEFAULT_VERBOSE

    • m_threadCount

      private int m_threadCount

    • m_parallelMode

      private XmlSuite.ParallelMode m_parallelMode

    • m_configFailurePolicy

      private XmlSuite.FailurePolicy m_configFailurePolicy

    • m_commandLineTestClasses

      private Class<?>[] m_commandLineTestClasses

    • m_defaultSuiteName

      private String m_defaultSuiteName

    • m_defaultTestName

      private String m_defaultTestName

    • m_methodDescriptors

      private final Map<String,Integer> m_methodDescriptors

    • m_selectors

      private final Set<XmlMethodSelector> m_selectors

    • DEFAULT_OBJECT_FACTORY

      private static final ITestObjectFactory DEFAULT_OBJECT_FACTORY

    • m_objectFactory

      private ITestObjectFactory m_objectFactory

    • m_invokedMethodListeners

      private final Map<Class<? extends IInvokedMethodListener>,IInvokedMethodListener> m_invokedMethodListeners

    • m_dataProviderThreadCount

      private Integer m_dataProviderThreadCount

    • m_jarPath

      private String m_jarPath

    • m_xmlPathInJar

      private String m_xmlPathInJar
      The path of the testng.xml file inside the jar file

    • m_stringSuites

      private List<String> m_stringSuites

    • m_hookable

      private IHookable m_hookable

    • m_configurable

      private IConfigurable m_configurable

    • m_end

      protected long m_end

    • m_start

      protected long m_start

    • m_alterSuiteListeners

      private final Map<Class<? extends IAlterSuiteListener>,IAlterSuiteListener> m_alterSuiteListeners

    • m_isInitialized

      private boolean m_isInitialized

    • isSuiteInitialized

      private boolean isSuiteInitialized

    • exitCodeListener

      private final ExitCodeListener exitCodeListener

    • exitCode

      private ExitCode exitCode

    • m_executionVisualisers

      private final Map<Class<? extends IExecutionVisualiser>,IExecutionVisualiser> m_executionVisualisers

    • m_verbose

      private Integer m_verbose
      If m_verbose gets set, it will override the verbose setting in testng.xml

    • m_defaultAnnoProcessor

      private final IAnnotationTransformer m_defaultAnnoProcessor

    • m_annotationTransformer

      private IAnnotationTransformer m_annotationTransformer

    • m_skipFailedInvocationCounts

      private Boolean m_skipFailedInvocationCounts

    • m_methodInterceptors

      private final List<IMethodInterceptor> m_methodInterceptors

    • m_testNames

      private List<String> m_testNames
      The list of test names to run from the given suite

    • m_ignoreMissedTestNames

      private boolean m_ignoreMissedTestNames

    • m_suiteThreadPoolSize

      private Integer m_suiteThreadPoolSize

    • m_randomizeSuites

      private boolean m_randomizeSuites

    • m_alwaysRun

      private boolean m_alwaysRun

    • m_preserveOrder

      private Boolean m_preserveOrder

    • m_groupByInstances

      private Boolean m_groupByInstances

    • m_generateResultsPerSuite

      private boolean m_generateResultsPerSuite

    • m_configuration

      private IConfiguration m_configuration

    • m_serviceLoaderClassLoader

      private URLClassLoader m_serviceLoaderClassLoader

    • serviceLoaderListeners

      private final Map<Class<? extends ITestNGListener>,ITestNGListener> serviceLoaderListeners

  • Constructor Details

    • TestNG

      public TestNG()
      Default constructor. Setting also usage of default listeners/reporters.

    • TestNG

      public TestNG(boolean useDefaultListeners)
      Used by maven2 to have 0 output of any kind come out of testng.
      Parameters:
      useDefaultListeners - Whether or not any default reports should be added to tests.

  • Method Details

    • init

      private void init(boolean useDefaultListeners)

    • toggleFailureIfAllTestsWereSkipped

      public void toggleFailureIfAllTestsWereSkipped(boolean failIfAllTestsSkipped)
      Parameters:
      failIfAllTestsSkipped - - Whether TestNG should enable/disable failing when all the tests were skipped and nothing was run (Mostly when a test is powered by a data provider and when the data provider itself fails causing all tests to skip).

    • setListenersToSkipFromBeingWiredInViaServiceLoaders

      public void setListenersToSkipFromBeingWiredInViaServiceLoaders(String... listeners)
      Parameters:
      listeners - - An array of fully qualified class names that should be skipped from being wired in via service loaders.

    • getStatus

      public int getStatus()

    • setOutputDirectory

      public void setOutputDirectory(String outputdir)
      Sets the output directory where the reports will be created.
      Parameters:
      outputdir - The directory.

    • setUseDefaultListeners

      public void setUseDefaultListeners(boolean useDefaultListeners)
      Parameters:
      useDefaultListeners - If true before run(), the default listeners will not be used.
      • org.testng.reporters.TestHTMLReporter
      • org.testng.reporters.JUnitXMLReporter
      • org.testng.reporters.XMLReporter
      See Also:

    • setTestJar

      public void setTestJar(String jarPath)
      Sets a jar containing a testng.xml file.
      Parameters:
      jarPath - - Path of the jar

    • setXmlPathInJar

      public void setXmlPathInJar(String xmlPathInJar)
      Parameters:
      xmlPathInJar - Sets the path to the XML file in the test jar file.

    • parseSuiteFiles

      private void parseSuiteFiles()

    • getProcessor

      private OverrideProcessor getProcessor()

    • parseSuite

      private Collection<XmlSuite> parseSuite(String suitePath)

    • processCommandLineArgs

      private Collection<XmlSuite> processCommandLineArgs(Collection<XmlSuite> allSuites)

    • processParallelModeCommandLineArgs

      private void processParallelModeCommandLineArgs(XmlSuite suite)

    • initializeSuitesAndJarFile

      public void initializeSuitesAndJarFile()

    • setThreadCount

      public void setThreadCount(int threadCount)
      Parameters:
      threadCount - Define the number of threads in the thread pool.

    • setParallel

      @Deprecated public void setParallel(String parallel)
      Deprecated.
      Use #setParallel(XmlSuite.ParallelMode) instead
      Parameters:
      parallel - Define whether this run will be run in parallel mode.

    • setParallel

      public void setParallel(XmlSuite.ParallelMode parallel)

    • setCommandLineSuite

      public void setCommandLineSuite(XmlSuite suite)

    • setTestClasses

      public void setTestClasses(Class[] classes)
      Set the test classes to be run by this TestNG object. This method will create a dummy suite that will wrap these classes called "Command Line Test".

      If used together with threadCount, parallel, groups, excludedGroups than this one must be set first.

      Parameters:
      classes - An array of classes that contain TestNG annotations.

    • splitMethod

      private String[] splitMethod(String m)
      Given a string com.example.Foo.f1, return an array where [0] is the class and [1] is the method.

    • createCommandLineSuitesForMethods

      private List<XmlSuite> createCommandLineSuitesForMethods(List<String> commandLineMethods)
      Parameters:
      commandLineMethods - a string with the form "com.example.Foo.f1,com.example.Bar.f2"
      Returns:
      a list of XmlSuite objects that represent the list of classes and methods passed in parameter.

    • createCommandLineSuitesForClasses

      private List<XmlSuite> createCommandLineSuitesForClasses(Class[] classes)

    • addMethodSelector

      public void addMethodSelector(String className, int priority)

    • addMethodSelector

      public void addMethodSelector(XmlMethodSelector selector)

    • setReportAllDataDrivenTestsAsSkipped

      public void setReportAllDataDrivenTestsAsSkipped(boolean reportAllDataDrivenTestsAsSkipped)

    • getReportAllDataDrivenTestsAsSkipped

      public boolean getReportAllDataDrivenTestsAsSkipped()

    • propagateDataProviderFailureAsTestFailure

      public void propagateDataProviderFailureAsTestFailure()

    • isPropagateDataProviderFailureAsTestFailure

      public boolean isPropagateDataProviderFailureAsTestFailure()

    • setTestSuites

      public void setTestSuites(List<String> suites)
      Set the suites file names to be run by this TestNG object. This method tries to load and parse the specified TestNG suite xml files. If a file is missing, it is ignored.
      Parameters:
      suites - A list of paths to one more XML files defining the tests. For example: 
       TestNG tng = new TestNG();
 List<String> suites = Lists.newArrayList();
 suites.add("c:/tests/testng1.xml");
 suites.add("c:/tests/testng2.xml");
 tng.setTestSuites(suites);
 tng.run();

    • setXmlSuites

      public void setXmlSuites(List<XmlSuite> suites)
      Specifies the XmlSuite objects to run.
      Parameters:
      suites - - The list of XmlSuite objects.
      See Also:

    • setExcludedGroups

      public void setExcludedGroups(String groups)
      Define which groups will be excluded from this run.
      Parameters:
      groups - A list of group names separated by a comma.

    • setGroups

      public void setGroups(String groups)
      Define which groups will be included from this run.
      Parameters:
      groups - A list of group names separated by a comma.

    • setTestRunnerFactoryClass

      private void setTestRunnerFactoryClass(Class<? extends ITestRunnerFactory> testRunnerFactoryClass)

    • setTestRunnerFactory

      protected void setTestRunnerFactory(ITestRunnerFactory itrf)

    • setObjectFactory

      public void setObjectFactory(Class<? extends ITestObjectFactory> c)

    • setObjectFactory

      public void setObjectFactory(ITestObjectFactory factory)

    • setListenerClasses

      public void setListenerClasses(List<Class<? extends ITestNGListener>> classes)
      Define which listeners to user for this run.
      Parameters:
      classes - A list of classes, which must be either ISuiteListener, ITestListener or IReporter

    • addListener

      @Deprecated public void addListener(Object listener)
      Deprecated.
      Use addListener(ITestNGListener) instead
      Parameters:
      listener - The listener to add

    • maybeAddListener

      private static <E> void maybeAddListener(Map<Class<? extends E>,E> map, E value)

    • maybeAddListener

      private static <E> void maybeAddListener(Map<Class<? extends E>,E> map, Class<? extends E> type, E value, boolean quiet)

    • addListener

      public void addListener(ITestNGListener listener)

    • getReporters

      public Set<IReporter> getReporters()

    • getTestListeners

      public List<ITestListener> getTestListeners()

    • getSuiteListeners

      public List<ISuiteListener> getSuiteListeners()

    • setVerbose

      public void setVerbose(int verbose)
      Sets the level of verbosity. This value will override the value specified in the test suites.
      Parameters:
      verbose - the verbosity level (0 to 10 where 10 is most detailed) Actually, this is a lie: you can specify -1 and this will put TestNG in debug mode (no longer slicing off stack traces and all).

    • setExecutorFactoryClass

      public void setExecutorFactoryClass(String clazzName)

    • setGenerateResultsPerSuite

      public void setGenerateResultsPerSuite(boolean generateResultsPerSuite)

    • createExecutorFactoryInstanceUsing

      private IExecutorFactory createExecutorFactoryInstanceUsing(String clazzName)

    • setExecutorFactory

      public void setExecutorFactory(IExecutorFactory factory)

    • getExecutorFactory

      public IExecutorFactory getExecutorFactory()

    • initializeCommandLineSuites

      private void initializeCommandLineSuites()

    • initializeCommandLineSuitesParams

      private void initializeCommandLineSuitesParams()

    • initializeCommandLineSuitesGroups

      private void initializeCommandLineSuitesGroups()

    • initializeCommandLineSuitesGroups

      private static void initializeCommandLineSuitesGroups(XmlSuite s, boolean hasIncludedGroups, String[] m_includedGroups, boolean hasExcludedGroups, String[] m_excludedGroups)

    • addReporter

      private void addReporter(Class<? extends IReporter> r)

    • initializeDefaultListeners

      private void initializeDefaultListeners()

    • initializeConfiguration

      private void initializeConfiguration()

    • addListeners

      private void addListeners(XmlSuite s)

    • addServiceLoaderListeners

      private void addServiceLoaderListeners()
      Using reflection to remain Java 5 compliant.

    • sanityCheck

      private void sanityCheck()
      Before suites are executed, do a sanity check to ensure all required conditions are met. If not, throw an exception to stop test execution
      Throws:
      TestNGException - if the sanity check fails

    • initializeEverything

      public void initializeEverything()
      Invoked by the remote runner.

    • run

      public void run()
      Run TestNG.

    • runSuites

      protected List<ISuite> runSuites()
      Run the test suites.

      This method can be overridden by subclass.
      For example, DistributedTestNG to run in master/slave mode according to commandline args.

      Returns:
      - List of suites that were run as ISuite objects.
      Since:
      6.9.11 when moving distributed/remote classes out into separate project

    • runSuiteAlterationListeners

      private void runSuiteAlterationListeners()

    • runExecutionListeners

      private void runExecutionListeners(boolean start)

    • usage

      private static void usage()

    • generateReports

      private void generateReports(List<ISuite> suiteRunners)

    • runSuitesLocally

      public List<ISuite> runSuitesLocally()
      This needs to be public for maven2, for now..At least until an alternative mechanism is found.
      Returns:
      The locally run suites

    • error

      private static void error(String s)

    • getVerbose

      private int getVerbose(XmlSuite xmlSuite)
      Returns:
      the verbose level, checking in order: the verbose level on the suite, the verbose level on the TestNG object, or 1.

    • runSuitesSequentially

      private void runSuitesSequentially(XmlSuite xmlSuite, SuiteRunnerMap suiteRunnerMap, int verbose, String defaultSuiteName)
      Recursively runs suites. Runs the children suites before running the parent suite. This is done so that the results for parent suite can reflect the combined results of the children suites.
      Parameters:
      xmlSuite - XML Suite to be executed
      suiteRunnerMap - Maps XmlSuites to respective ISuite
      verbose - verbose level
      defaultSuiteName - default suite name

    • populateSuiteGraph

      private void populateSuiteGraph(IDynamicGraph<ISuite> suiteGraph, SuiteRunnerMap suiteRunnerMap, XmlSuite xmlSuite)
      Populates the dynamic graph with the reverse hierarchy of suites. Edges are added pointing from child suite runners to parent suite runners, hence making parent suite runners dependent on all the child suite runners
      Parameters:
      suiteGraph - dynamic graph representing the reverse hierarchy of SuiteRunners
      suiteRunnerMap - Map with XMLSuite as key and its respective SuiteRunner as value
      xmlSuite - XML Suite

    • createSuiteRunners

      private void createSuiteRunners(SuiteRunnerMap suiteRunnerMap, XmlSuite xmlSuite)
      Creates the SuiteRunners and populates the suite runner map with this information
      Parameters:
      suiteRunnerMap - Map with XMLSuite as key and it's respective SuiteRunner as value. This is updated as part of this method call
      xmlSuite - Xml Suite (and its children) for which SuiteRunners are created

    • createSuiteRunner

      private SuiteRunner createSuiteRunner(XmlSuite xmlSuite)
      Creates a suite runner and configures its initial state

    • getConfiguration

      protected IConfiguration getConfiguration()

    • main

      public static void main(String[] argv)
      The TestNG entry point for command line execution.
      Parameters:
      argv - the TestNG command line parameters.

    • privateMain

      public static TestNG privateMain(String[] argv, ITestListener listener)
      Note: this method is not part of the public API and is meant for internal usage only.
      Parameters:
      argv - The param arguments
      listener - The listener
      Returns:
      The TestNG instance

    • configure

      protected void configure(CommandLineArgs cla)
      Configure the TestNG instance based on the command line parameters.
      Parameters:
      cla - The command line parameters

    • setIgnoreMissedTestNames

      private void setIgnoreMissedTestNames(boolean ignoreMissedTestNames)

    • setSuiteThreadPoolSize

      public void setSuiteThreadPoolSize(Integer suiteThreadPoolSize)

    • getSuiteThreadPoolSize

      public Integer getSuiteThreadPoolSize()

    • setRandomizeSuites

      public void setRandomizeSuites(boolean randomizeSuites)

    • alwaysRunListeners

      public void alwaysRunListeners(boolean alwaysRun)

    • setSourcePath

      @Deprecated public void setSourcePath(String path)
      Deprecated.
      This method is invoked by Maven's Surefire, only remove it once Surefire has been modified to no longer call it.
      Parameters:
      path - The path

    • parseInt

      private static int parseInt(Object value)

    • configure

      @Deprecated public void configure(Map cmdLineArgs)
      Deprecated.
      use new configure(CommandLineArgs) method
      This method is invoked by Maven's Surefire to configure the runner, do not remove unless you know for sure that Surefire has been updated to use the new configure(CommandLineArgs) method.
      Parameters:
      cmdLineArgs - The command line

    • setTestNames

      public void setTestNames(List<String> testNames)
      Parameters:
      testNames - Only run the specified tests from the suite.

    • setSkipFailedInvocationCounts

      public void setSkipFailedInvocationCounts(Boolean skip)

    • addReporter

      private void addReporter(ReporterConfig reporterConfig)

    • newReporterInstance

      private IReporter newReporterInstance(ReporterConfig config)
      Creates a reporter based on the configuration

    • setJUnit

      public void setJUnit(Boolean isJUnit)
      Specify if this run should be made in JUnit mode
      Parameters:
      isJUnit - - Specify if this run should be made in JUnit mode

    • setMixed

      public void setMixed(Boolean isMixed)
      Parameters:
      isMixed - Specify if this run should be made in mixed mode

    • validateCommandLineParameters

      protected static void validateCommandLineParameters(CommandLineArgs args)
      Double check that the command line parameters are valid.
      Parameters:
      args - The command line to check

    • hasFailure

      public boolean hasFailure()
      Returns:
      true if at least one test failed.

    • hasFailureWithinSuccessPercentage

      public boolean hasFailureWithinSuccessPercentage()
      Returns:
      true if at least one test failed within success percentage.

    • hasSkip

      public boolean hasSkip()
      Returns:
      true if at least one test was skipped.

    • exitWithError

      static void exitWithError(String msg)

    • getOutputDirectory

      public String getOutputDirectory()

    • getAnnotationTransformer

      public IAnnotationTransformer getAnnotationTransformer()

    • setAnnotationTransformer

      private void setAnnotationTransformer(IAnnotationTransformer t)

    • getDefaultSuiteName

      public String getDefaultSuiteName()
      Returns:
      the defaultSuiteName

    • setDefaultSuiteName

      public void setDefaultSuiteName(String defaultSuiteName)
      Parameters:
      defaultSuiteName - the defaultSuiteName to set

    • getDefaultTestName

      public String getDefaultTestName()
      Returns:
      the defaultTestName

    • setDefaultTestName

      public void setDefaultTestName(String defaultTestName)
      Parameters:
      defaultTestName - the defaultTestName to set

    • setConfigFailurePolicy

      public void setConfigFailurePolicy(XmlSuite.FailurePolicy failurePolicy)
      Sets the policy for whether or not to ever invoke a configuration method again after it has failed once. Possible values are defined in XmlSuite. The default value is XmlSuite.FailurePolicy.SKIP
      Parameters:
      failurePolicy - the configuration failure policy

    • getConfigFailurePolicy

      public XmlSuite.FailurePolicy getConfigFailurePolicy()
      Returns the configuration failure policy.
      Returns:
      config failure policy

    • getDefault

      @Deprecated public static TestNG getDefault()
      Deprecated.
      since 5.1
      Returns:
      The default instance

    • setConfigurable

      private void setConfigurable(IConfigurable c)

    • setHookable

      private void setHookable(IHookable h)

    • setMethodInterceptor

      public void setMethodInterceptor(IMethodInterceptor methodInterceptor)

    • setDataProviderThreadCount

      public void setDataProviderThreadCount(int count)

    • addClassLoader

      public void addClassLoader(ClassLoader loader)
      Add a class loader to the searchable loaders.
      Parameters:
      loader - The class loader to add

    • setPreserveOrder

      public void setPreserveOrder(boolean b)

    • getStart

      protected long getStart()

    • getEnd

      protected long getEnd()

    • setGroupByInstances

      public void setGroupByInstances(boolean b)

    • setServiceLoaderClassLoader

      public void setServiceLoaderClassLoader(URLClassLoader ucl)

    • addServiceLoaderListener

      private void addServiceLoaderListener(ITestNGListener l)

    • getServiceLoaderListeners

      public List<ITestNGListener> getServiceLoaderListeners()

    • setInjectorFactory

      public void setInjectorFactory(IInjectorFactory factory)