# `mix test.json`
[🔗](https://github.com/ZenHive/ex_unit_json/blob/main/lib/mix/tasks/test_json.ex#L1)

Runs tests and outputs results as JSON.

This task wraps `mix test` and configures ExUnit to use
`ExUnitJSON.Formatter` for JSON output instead of the default
CLI formatter.

## Setup

Add this to your project's `mix.exs` to ensure `test.json` runs in the test environment:

    def cli do
      [preferred_envs: ["test.json": :test]]
    end

This is required because Mix doesn't inherit `preferred_envs` from dependencies.

## Usage

    mix test.json
    mix test.json test/my_test.exs
    mix test.json test/my_test.exs:42

## Options

All standard `mix test` options are supported, plus:

  * `--summary-only` - Output only the summary, omit individual test results
  * `--all` - Output all tests (default shows only failures)
  * `--failures-only` - Output only failed tests (DEFAULT - AI-optimized)
  * `--first-failure` - Output only the first failed test (quick iteration)
  * `--filter-out PATTERN` - Mark failures matching pattern as filtered (can repeat)
  * `--output FILE` - Write JSON to file instead of stdout
  * `--compact` - JSONL output with minimal fields (one line per test)
  * `--group-by-error` - Group failures by similar error message
  * `--quiet` - Suppress Logger output and TIP warnings for clean JSON piping
  * `--no-warn` - Suppress the "use --failed" warning when previous failures exist
  * `--no-retry` - Disable automatic retry of failed tests (see "Automatic Retry")
  * `--cover` - Enable code coverage (off by default for faster runs)
  * `--cover-threshold N` - Fail if overall coverage is below N (0-100). Requires `--cover`

## Coverage

Coverage is disabled by default for faster test runs. Use `--cover` to enable:

    mix test.json --cover

The JSON output includes a `coverage` key with:

    "coverage": {
      "total_percentage": 96.96,
      "total_lines": 330,
      "covered_lines": 320,
      "modules": [
        {
          "module": "MyApp.Module",
          "file": "lib/my_app/module.ex",
          "percentage": 92.68,
          "covered_lines": 38,
          "uncovered_lines": [45, 67, 89]
        }
      ]
    }

The `threshold` and `threshold_met` fields are added only when
`--cover-threshold` is used. Coverage cannot be combined with `--compact`
(a warning is printed and coverage data is omitted).

## Default Behavior (v0.3.0+)

By default, `mix test.json` outputs only failed tests (equivalent to `--failures-only`).
This is optimized for AI agents where passing tests are noise.

Use `--all` to include all tests in output when needed.

## Flag Precedence

When multiple filtering flags are combined, they follow this priority:

  1. `--summary-only` - Highest priority, omits tests array entirely
  2. `--first-failure` - Returns only the first failed test
  3. `--failures-only` / `--all` - Filter to failures or show all tests

For example, `--summary-only --all` will omit the tests array.

## Iteration Workflow

When previous test failures exist, a tip is shown suggesting
to use `--failed` for faster iteration:

    TIP: 3 previous failure(s) exist. Consider:
      mix test.json --failed

This warning is skipped when:
  * `--failed` is already used
  * A specific file or directory is targeted
  * `--only` or `--exclude` tag filters are used
  * `--no-warn` or `--quiet` is passed
  * Auto-retry is enabled (the retry supersedes the manual `--failed` hint)

## Automatic Retry (v0.5.0+)

By **default**, when a run has failures, `mix test.json` automatically re-runs
only the previously-failed tests (ExUnit's native `--failed`) in a subprocess,
then merges the two runs to distinguish:

  * **confirmed** failures — failed both runs. Stay in `tests`, stay red.
  * **flaky** failures — failed run 1, passed run 2. Surfaced in a top-level
    `flaky` array (never hidden) but no longer block the run.

When every first-run failure heals on retry, the suite reports
`summary.result == "passed"` and exits 0, so an AI agent isn't blocked by a
flake — while the flaky tests are still named in the output. A test that fails
both runs stays a hard failure (exit non-zero).

Tests invalidated by a flaky `setup_all` failure are handled the same way:
when the module heals on retry, its tests resolve to their retry state
(passed or failed) instead of staying `"invalid"`.

The merged output adds (only when a retry ran): a `flaky` array, a
`summary.flaky` count, and a `retry` metadata block. The schema `version`
stays `1` (additive).

Auto-retry is **skipped** (run-1 output is reported unchanged) when it would be
meaningless or unsupported: `--no-retry`, `config :ex_unit_json, retry: false`,
`--failed` (already iterating), `--summary-only`, `--first-failure`,
`--compact`, `--group-by-error`, `--filter-out`, a `file:line` target, or an
umbrella project.

Disable it entirely:

    mix test.json --no-retry

    # or in config/test.exs
    config :ex_unit_json, retry: false

## Strict Enforcement

To block full test runs when failures exist (useful for AI-assisted workflows):

    # config/test.exs
    config :ex_unit_json, enforce_failed: true

This will exit with an error instead of just warning.

## Examples

    # Run all tests with JSON output
    mix test.json

    # Run specific file
    mix test.json test/my_test.exs

    # Output only failures to a file
    mix test.json --failures-only --output failures.json

---

*Consult [api-reference.md](api-reference.md) for complete listing*