External Tools

cargo-criterion provides a machine-readable output stream which other tools can consume to collect information about the Criterion.rs benchmarks.

To enable this output stream, pass the --message-format argument when running cargo-criterion.

JSON messages

When passing --message-format=json cargo-criterion will output information about:

• Benchmarks, including the basic statistics about the measurements
• Benchmark groups

The output goes to stdout, with one JSON object per line. The reason field distinguishes different kinds of messages.

Additional messages or fields may be added to the output in the future.

Benchmark Complete Messages

The "benchmark-complete" message includes the measurements and basic statistics from a single Criterion.rs benchmark. The message format is as follows:

{
  /* The "reason" indicates which kind of message this is. */
  "reason": "benchmark-complete",
  /* The id is the identifier of this benchmark */
  "id": "norm",
  /* Path to the directory containing the report for this benchmark */
  "report_directory": "target/criterion/reports/norm",
  /* List of integer iteration counts */
  "iteration_count": [
    30,
    /* ... */
    3000
  ],
  /* List of floating point measurements (eg. time, CPU cycles) taken 
  from the benchmark */
  "measured_values": [
    124200.0,
    /* ... */
    9937100.0
  ],
  /* The unit associated with measured_values. */
  "unit": "ns",
  /* The throughput value associated with this benchmark. This can be used 
  to calculate throughput rates, eg. in bytes or elements per second. */
  "throughput": [
    {
      "per_iteration": 1024,
      "unit": "elements"
    }
  ],
  /* Confidence intervals for the basic statistics that cargo-criterion 
  computes. */
  /* 
  "typical" is either the slope (if available) or the mean (if not). It
  makes a good general-purpose estimate of the typical performance of a
  function.
  */
  "typical": {
    "estimate": 3419.4923993891925,
    "lower_bound": 3375.24221103098,
    "upper_bound": 3465.458469579234,
    "unit": "ns"
  },
  "mean": {
    "estimate": 3419.5340743105917,
    "lower_bound": 3374.4765622217083,
    "upper_bound": 3474.096214164006,
    "unit": "ns"
  },
  "median": {
    "estimate": 3362.8249818445897,
    "lower_bound": 3334.259259259259,
    "upper_bound": 3387.5146198830407,
    "unit": "ns"
  },
  "median_abs_dev": {
    "estimate": 130.7846461816652,
    "lower_bound": 96.55619525548211,
    "upper_bound": 161.1643711235156,
    "unit": "ns"
  },
  
  /* Note that not all benchmarks can measure the slope, so it may be 
  missing. */
  "slope": {
    "estimate": 3419.4923993891925,
    "lower_bound": 3375.24221103098,
    "upper_bound": 3465.458469579234,
    "unit": "ns"
  },

  /* "change" contains some additional statistics about the difference 
  between this run and the last */
  "change": {
    /* Percentage differences in the mean & median values */
    "mean": {
      "estimate": 0.014278477848724602,
      "lower_bound": -0.01790259435189548,
      "upper_bound": 0.03912764721581533,
      "unit": "%"
    },
    "median": {
      "estimate": 0.012211662837601445,
      "lower_bound": -0.0005448009516478807,
      "upper_bound": 0.024243170768727857,
      "unit": "%"
    },
    /* 
    Indicates whether cargo-criterion found a statistically-significant 
    change. Values are NoChange, Improved, or Regressed
    */
    "change": "NoChange"
  }
}

Group Complete Messages

When a benchmark group is completed, cargo-criterion emits a "group-complete" message containing some information about the group.

{
  "reason": "group-complete",
  /* The name of the benchmark group */
  "group_name": "throughput",
  /* List of the benchmark IDs in this group */
  "benchmarks": [
    "throughput/Bytes",
    "throughput/Bytes",
    "throughput/Elem"
  ],
  /* Path to the directory that contains the report for this group */
  "report_directory": "target/criterion/reports/throughput"
}