Reporting

Cucumber uses reporter plugins to produce reports that contain information about what scenarios have passed or failed.

Some plugins are built-in, others have to be installed separately. You can also build your own.

This page documents built-in formatter plugins as well common third-party plugins. Available plugins may vary per programming language.

Built-in reporter plugins

There are several reporter plugins built into Cucumber:

  • progress
  • pretty
  • html
  • etc.

Cucumber Pro plugin

This plugin publishes results to Cucumber Pro.

Requirements

Your project must be stored in a Git repository, and you must be using one of the following CI servers:

Installation

Configuration

Create a file called cucumber.yml in the root directory of your git repository with the following contents (replace *** with actual values).

cucumberpro:
  # The name of the Cucumber Pro project.
  # You can leave this blank if your Cucumber Pro project name is identical to the
  # CI project name, and you you're using one of Bamboo, Circle CI, TFS, Travis
  # or Wercker.
  projectname: ***
  # The URL of the Cucumber Pro server.
  # You can leave this blank if you are publishing to https://app.cucumber.pro/
  url: ***

Authentication

If you are publishing to https://app.cucumber.pro/ you also have to define an environment variable named CUCUMBERPRO_TOKEN in your CI project. The value should be your Cucumber Pro project’s API token, avaiable from the project settings page. Consult your CI server’s documentation for details about how to define environment variables.

Authentication is not required on a privately hosted Cucumber Pro Appliance.

Activation

The plugin will activate itself automatically if it detects that it’s running in one of the supported CI environments. When you run from your workstation the plugin will not be activated, and will not publish results.

When you configure the plugin for the first time you can force-activate the plugin from your work station by defining the following environment variables:

  • GIT_COMMIT - you can find it by running git rev-parse HEAD
  • GIT_BRANCH - you can find it by running git rev-parse --abbrev-ref HEAD

This is useful for verifying that you have configured the plugin correctly.

Profiles

If you run several times as part of your build (with different options, perhaps different tags), you can specify a different profile name for each run. This allows Cucumber Pro to show separate results for each profile.

The profile name can be specified in the CUCUMBERPRO_PROFILE environment variable, which you would typically define in a wrapper script that launches .

Advanced configuration

The cucumber.yml configuration file has more options that can be overridden for finer grained control. The defaults are as follows:

cucumberpro:
  # The name of the Cucumber Pro project.
  projectname:

  # The project-specific authentication token. You can find it in the project settings (press `?` to display it).
  #
  # Rather than defining the value in this file we recommend defining the `CUCUMBERPRO_TOKEN` environment variable
  # in your CI server.
  #
  # Consult your CI server's documentation for details about defining per-project environment variables.
  # Some CI servers such as Travis and Circle CI allow you to define environment variables in a file checked into git.
  # *DO NOT DO THIS* - as it would allow anyone with read acceess to your repository to publish results.
  token:

  # The plugin sends your local environment variables to Cucumber Pro so it can detect the CI build number,
  # git branch/tag and other information about the build. This mask is a regular expression for filtering
  # out sensitive values that should not be sent to Cucumber Pro.
  envmask: SECRET|KEY|TOKEN|PASSWORD|PWD

  # Sets the log level to one of `DEBUG`, `INFO`, `WARN`, `ERROR` or `FATAL`. Defaults to `WARN`.
  # Setting it to `DEBUG` will also print the current configuration when the plugin runs.
  logging: INFO

  # Writes out the log messages to the specified file. Useful for debugging.
  logfile:

  # Override this if you are using a privately hosted Cucumber Pro Appliance.
  # We recommend setting this with a CUCUMBERPRO_URL environment variable defined globally on your build server.
  url: https://app.cucumber.pro/

  connection:
    # Set this to false if you want the build to break in case Cucumber Pro is unavailable.
    ignoreerror: true

    # If the http connection for publishing results takes longer than this (milliseconds),
    # time out the connection.
    timeout: 5000

You can make some of the settings global by creating a file with global settings. The plugin will load the configuration in all the following files (if they exist):

Every setting can also be overridden with environment variables. For example, if you want the plugin to log more verbosely:

# Linux / MacOS
export CUCUMBERPRO_LOGGING=DEBUG

# Windows
SET CUCUMBERPRO_LOGGING=DEBUG

Third-party plugins

There are also many third-party plugins:

  • Masterthought
  • TeamCity - prints Cucumber results in a format for interpretation by a TeamCity build agent.
  • TextmateFormatter prints Cucumber results as HTML with enhanced styling and Javascript for Textmate (Included in the Cucumber core since 0.4.5)
  • SlowHandCuke - Simple tweak to the Pretty formatter to display the currently running Step as it is running
  • timestamped-scenarios - Append test run timestamps to each Scenario name as it is being output.
  • Fivemat - Cucumber formatter that gives each test file its own line of dots.
  • Fuubar - The insta-failing progress bar formatter
  • Viewcumber - Cucumber formatter which generates an HTML website to browse your Scenarios and view screen capture of every single Step.
  • cucumber_timing_presenter - formatter that calculates timing metrics as well as two graphs showing impact of Step time on overall build time.
  • Bilgerat - formatter that sends failure messages to HipChat rooms.
  • cucumber_statistics - Tracks timing and displays results in a single HTML page with outliers highlighted in a table sortable by various metrics.
  • cucumber_characteristics - Generates HTML/JSON reports on overall test timings, as well as timings and usage of Steps, Features, and Examples. Also lists unused and ambiguous (Cucumber 1.x) Steps. Compatible with Cucumber 1.x and 2.1+ and Ruby 1.9+.
  • allure-cucumber - Allure adaptor for Cucumber. This formatter generates the XML files for Allure reporting framework.

You can help us improve this documentation. Edit this page.