Reporting

Cucumber uses reporter plugins to produce reports that contain information about what scenarios have passed or failed.

Some plugins are built-in, others have to be installed separately. You can also build your own.

This page documents built-in formatter plugins, the Cucumber Pro plugin, and some common third-party plugins. Available plugins may vary per programming language.

Built-in reporter plugins

There are several reporter plugins built into Cucumber:

  • progress
  • pretty
  • html
  • json
  • rerun
  • junit

Cucumber Pro plugin

This plugin publishes results to Cucumber Pro.

Requirements

Your project must be stored in a Git repository, and you must be using one of the following CI servers:

Installation

Configuration

Create a file called cucumber.yml in the root directory of your git repository with the following contents (replace *** with actual values).

cucumberpro:
  # The name of the Cucumber Pro project.
  # You can leave this blank if your Cucumber Pro project name is identical to the
  # CI project name, and you you're using one of Bamboo, Circle CI, TFS, Travis
  # or Wercker.
  projectname: ***
  # The URL of the Cucumber Pro server.
  # You can leave this blank if you are publishing to https://app.cucumber.pro/
  url: ***

Authentication

If you are publishing to https://app.cucumber.pro/ you also have to define an environment variable named CUCUMBERPRO_TOKEN in your CI project. The value should be your Cucumber Pro project’s API token, avaiable from the project settings page. Consult your CI server’s documentation for details about how to define environment variables.

Authentication is not required on a privately hosted Cucumber Pro Appliance.

Activation

The plugin will activate itself automatically if it detects that it’s running in one of the supported CI environments. When you run from your workstation the plugin will not be activated, and will not publish results.

When you configure the plugin for the first time you can force-activate the plugin from your work station by defining the following environment variables:

  • GIT_COMMIT - you can find it by running git rev-parse HEAD
  • GIT_BRANCH - you can find it by running git rev-parse --abbrev-ref HEAD

This is useful for verifying that you have configured the plugin correctly.

Profiles

If you run several times as part of your build (with different options, perhaps different tags), you can specify a different profile name for each run. This allows Cucumber Pro to show separate results for each profile.

The profile name can be specified in the CUCUMBERPRO_PROFILE environment variable, which you would typically define in a wrapper script that launches .

Advanced configuration

The cucumber.yml configuration file has more options that can be overridden for finer grained control. The defaults are as follows:

cucumberpro:
  # The name of the Cucumber Pro project.
  projectname:

  # The project-specific authentication token. You can find it in the project settings (press `?` to display it).
  #
  # Rather than defining the value in this file we recommend defining the `CUCUMBERPRO_TOKEN` environment variable
  # in your CI server.
  #
  # Consult your CI server's documentation for details about defining per-project environment variables.
  # Some CI servers such as Travis and Circle CI allow you to define environment variables in a file checked into git.
  # *DO NOT DO THIS* - as it would allow anyone with read acceess to your repository to publish results.
  token:

  # The plugin sends your local environment variables to Cucumber Pro so it can detect the CI build number,
  # git branch/tag and other information about the build. This mask is a regular expression for filtering
  # out sensitive values that should not be sent to Cucumber Pro.
  envmask: SECRET|KEY|TOKEN|PASSWORD|PWD

  # Sets the log level to one of `DEBUG`, `INFO`, `WARN`, `ERROR` or `FATAL`. Defaults to `WARN`.
  # Setting it to `DEBUG` will also print the current configuration when the plugin runs.
  logging: INFO

  # Writes out the log messages to the specified file. Useful for debugging.
  logfile:

  # Override this if you are using a privately hosted Cucumber Pro Appliance.
  # We recommend setting this with a CUCUMBERPRO_URL environment variable defined globally on your build server.
  url: https://app.cucumber.pro/

  connection:
    # Set this to false if you want the build to break in case Cucumber Pro is unavailable.
    ignoreerror: true

    # If the http connection for publishing results takes longer than this (milliseconds),
    # time out the connection.
    timeout: 5000

You can make some of the settings global by creating a file with global settings. The plugin will load the configuration in all the following files (if they exist):

Every setting can also be overridden with environment variables. For example, if you want the plugin to log more verbosely:

# Linux / MacOS
export CUCUMBERPRO_LOGGING=DEBUG

# Windows
SET CUCUMBERPRO_LOGGING=DEBUG

Custom formatters

The Cucumber formatter API is readily extensible. A formatter can be any class implementing the event-based formatter API. To use your custom formatter, run Cucumber using the --format flag:

cucumber --format CustomFormatter

Formatter API

Cucumber uses an event-based API for its formatters. These formatters respond to several defined events, with event handlers defined in the formatter’s constructor.

Configuration object

The formatter initializer is passed a object. This is the configuration for the test run, including default configurations and options passed in at the command line. It can be useful to access these options, so that your formatter can modify its behavior in response to user directives.

Event objects

Every time an event is fired, an event object is passed to the corresponding handler. Each event object belongs to a class corresponding to its name.

For instance, the gherkin_source_read event creates a Cucumber::Events::GherkinSourceRead object.

Each of these objects provides a different API to access relevant data such as Gherkin source, feature and step names, and passed/failed status of a step or test case.

Note: Following are the event objects for Cucumber Ruby; other programming languages might have slightly different events.

gherkin_source_read

The gherkin_source_read event is fired after the contents of a new feature file have been read. It has two attributes: * path: The file path of the feature file that has been read * body: The raw Gherkin contained in that file, as a string

step_definition_registered

The step_definition_registered event is fired after each step definition has been registered. It has one attribute: * step_definition: The Cucumber::Glue::StepDefinition object that has just been registered. This object responds to the following instance methods: - #backtrace_line: The file, line, and step definition that will appear in a backtrace, as in: features/step_definitions/cuke_steps.rb:5:in "this step passes" - #expression: The expression used in the step definition, as in "I have {int} cukes in my belly" (when a Cucumber expression is used) or /^I have (\d+) cukes in my belly$/ (when a regular expression is used) - #file: The file where the step definition occurs, as a string, as in "features/step_definitions/cuke_steps.rb" - #file_colon_line: The file and line where the step definition occurs, as a string, as in "features/step_definitions/cuke_steps.rb:5" - #location: Alias of #file_colon_line - #to_hash: A hash of data about the step definition, such as {:source=>{:type=>"cucumber expression", :expression=>"I fail"}, :regexp=>{:source=>"^I fail$", :flags=>""}}

test_run_started

The test_run_started event is fired at the beginning of the test run. It has one attribute: * test_cases: The test cases included in the test run. This is an array of Cucumber::Core::Test::Case objects. Each of these objects responds to the following instance methods: - #all_locations: Returns an array with the location of each element (feature, scenario, and step) in the feature file for the test case. - #all_source: Returns an array of the features, scenarios, and steps in the test case, as strings, without the keywords - #around_hooks: Returns an array of the Around hooks associated with that test case - #feature: Returns the name of the feature for the test case - #keyword: Returns the Gherkin keyword associated with the test case ("Feature", "Scenario", etc.) - #language: Returns the language indicated for the test case, as a Gherkin::Dialect object - #location: Returns the location of the first scenario in the test case - #name: Returns the name of the feature for the test case - #source: Returns the names of the features and scenarios in the test case, as strings, without the keywords - #step_count: Returns the integer number of steps in the test case - #tags: Returns an array of tags for the test case - #test_steps: Returns an array of the steps in the test case, without the keywords

test_run_finished

The test_run_finished event is fired after the test run has finished. It has no additional attributes or methods defined beyond those defined on the base event class (i.e., #attributes, #event_id and #to_h), meaning you cannot access test cases, Gherkin source, etc. from the handler for this event. This event can be used, for example, to print the running time of the test suite.

test_case_started

The test_case_started event is fired when a Cucumber::Core::Test::Case is about to be executed. It has one attribute: * test_case: The Cucumber::Core::Test::Case object (described above) that is about to be executed.

test_case_finished

The test_case_finished event is fired when a Cucumber::Core::Test::Case has finished executing. It has two attributes: * test_case: The Cucumber::Core::Test::Case object (described above) that has just finished * result: The result (passed, failed, pending, skipped) of the test case

step_activated

The step_activated event is fired when a test step has been activated. It has two attributes: * test_step: The Cucumber::Core::Test::Step object (described above) that has been activated * step_match: A Cucumber::StepMatch object. This object responds to the following instance methods: - #args: The arguments passed to the step - #backtrace_line: The backtrace line from the step definition including the file, line, and step expression - #format_args: The text of the step - #location: The step definition file where the step is defined - #step_arguments: The arguments passed to the step - #step_definition: The Cucumber::Glue::StepDefinition object (described above) corresponding to the step - #text_length: The number of characters in the step text

test_step_started

The test_step_started event is fired just before each Cucumber::Core::Test::Step is started. It has one attribute: * test_step: The Cucumber::Core::Test::Step object that has just started. This object responds to the following methods that may be useful for the purposes of your formatter: - #action_location: The feature file in which the step appears and the location of its scenario within that file, in the form features/file.feature:line - #location: The feature file in which the step appears and the location of the step within that file, in the same form as described under #action_location - #source: The Gherkin source of the feature file in which the step appears, without keywords - #text: The text of the step, without the Gherkin keyword

test_step_finished

The test_step_finished event is fired after each Cucumber::Core::Test::Step has finished running. It has two attributes: * test_step: The Cucumber::Core::Test::Step object (described above) that has just executed * result The result (passed, failed, pending, skipped) of the test case

Third-party plugins

There are also many third-party plugins:

  • Masterthought
  • TeamCity - prints Cucumber results in a format for interpretation by a TeamCity build agent.
  • TextmateFormatter prints Cucumber results as HTML with enhanced styling and Javascript for Textmate (Included in the Cucumber core since 0.4.5)
  • SlowHandCuke - Simple tweak to the Pretty formatter to display the currently running Step as it is running
  • timestamped-scenarios - Append test run timestamps to each Scenario name as it is being output.
  • Fivemat - Cucumber formatter that gives each test file its own line of dots.
  • Fuubar - The insta-failing progress bar formatter
  • Viewcumber - Cucumber formatter which generates an HTML website to browse your Scenarios and view screen capture of every single Step.
  • cucumber_timing_presenter - formatter that calculates timing metrics as well as two graphs showing impact of Step time on overall build time.
  • Bilgerat - formatter that sends failure messages to HipChat rooms.
  • cucumber_statistics - Tracks timing and displays results in a single HTML page with outliers highlighted in a table sortable by various metrics.
  • cucumber_characteristics - Generates HTML/JSON reports on overall test timings, as well as timings and usage of Steps, Features, and Examples. Also lists unused and ambiguous (Cucumber 1.x) Steps. Compatible with Cucumber 1.x and 2.1+ and Ruby 1.9+.
  • allure-cucumber - Allure adaptor for Cucumber. This formatter generates the XML files for Allure reporting framework.
  • Cluecumber - Maven plugin for clear and concise Cucumber reporting.

You can help us improve this documentation. Edit this page.