Running at scale

One run, one build

A build is not only an HTML file. Writing one means rotating the previous output.json into archive/, writing a new output.json, appending a point to the trend chart, applying archive retention, recomputing Suite Highlights, and re-reading every retained build to rebuild the Analytics histories.

So four matrix legs each writing a report into one folder do not add up to a build. They overwrite each other's output.json and manufacture four builds out of one run: four trend points, four archive files, four Analytics entries, and Suite Highlights counting every suite four times — with nothing left on disk afterwards to say which four belonged together.

There are two ways a run can be spread out, and they get two different answers.

• One machine, many processesThat is pytest-xdist. The workers are children of one controller and share its filesystem, so they hand their records back in-process and the controller renders once. Nothing to configure.
• Many machines, or many separate pytest invocationsThat is sharding. There is no shared process, so each leg writes a bundle and no report at all, and a separate command merges the bundles later.

Why the merge works on records and nothing else

Two simpler designs were rejected in both directions, and the reasons are what decide the file formats.

• Merging output.json files does not work. That file keeps only status, message, test_name, rerun and duration per test — no node id, no logs, no steps, no attachments, no phases. A merge built on it would produce correct totals over an empty report, and could not name a single test in a JUnit file.
• Merging rendered HTML does not work. There is nothing to merge: the page is one document with one dashboard, one trend chart and one Analytics view, all computed from the records.

So the merge happens at the level of records — the dicts the plugin builds as each test finishes — and nowhere else.

What the report works out about the run

A report is a build artifact: it is read a week later, by somebody who cannot re-run it and cannot ask the machine anything. The panel could already name the host, the Python and the pytest — the easy half — while the half that explains a red build was left to whoever remembered to write a --build-info flag before the run.

The rows, and where each comes from

The systems detected by name

GitHub Actions, GitLab CI, CircleCI, Buildkite, Azure Pipelines, Travis CI, AppVeyor, Drone, Bitbucket Pipelines, Semaphore, AWS CodeBuild, TeamCity and Jenkins are each named individually, from the variable that actually identifies them — GITHUB_ACTIONS, GITLAB_CI, CIRCLECI, TF_BUILD, JENKINS_URL and so on. Jenkins is tested last on purpose: BUILD_NUMBER and BUILD_URL are set by several of the others, and JENKINS_URL is what actually says Jenkins. Anything else that sets CI or CONTINUOUS_INTEGRATION is still recorded as a build agent rather than passed off as somebody's laptop.

• A published build url is used as givenGitLab, Jenkins, CircleCI, Buildkite, Travis and Drone each publish their own, so a self-hosted install, a reverse proxy or a renamed domain is linked correctly rather than to a page on the vendor's hosting that does not exist. The rest are assembled from the documented variables.
• A re-run GitHub Actions workflow links its own attemptGITHUB_RUN_ID deliberately does not change on a re-run, so the bare run url opens the latest attempt — a report written by attempt 1 would have linked somebody else's failure.
• CI before git, for the revisionA CI checkout is a detached HEAD, where git itself answers HEAD — true and useless — while the system knows the branch somebody clicked. A folder that is not a checkout gets no such rows rather than a traceback.
• Detection never overrules youA branch, commit, ci or pipeline already named through --build-info or the build_info ini key is the answer shown, and the detected one is dropped rather than rendered beside it disagreeing.
• Workers are counted, not requestedThe row says 3 of 8 requested when the two differ: xdist quietly runs fewer workers than asked for when there are fewer tests than workers, and a report claiming eight when three ran reads as five crashes. A serial run gets no row at all.

The CI shape of a run, written down

A workflow file is a bad home for a report's settings. It is edited by whoever is fixing the pipeline, it is not what anybody runs locally, and the day the two drift apart is the day somebody opens a CI report and finds it missing the logs they went there for. A profile puts both shapes in one file, side by side, where the difference between them is the thing you can see:

[tool.pytest-html-reporter]
title = "PAYMENTS"

[tool.pytest-html-reporter.profiles.local]
open = "auto"
logs = "all"
archive_count = 10

[tool.pytest-html-reporter.profiles.ci]
open = "none"
logs = "failed"
attachments = "failed"
screenshots = "failed"
junit = "report/junit.xml"
archive_days = 30

- name: Run the suite
  run: |
    pytest tests/ -n auto \
      --report-profile=ci \
      --build-info branch="$GITHUB_REF_NAME" \
      --build-info commit="$GITHUB_SHA" \
      --build-info run="$GITHUB_RUN_ID"

$ pytest tests/ --report-profile=local

The pieces a workflow legitimately owns — the branch, the commit, the run id — stay on the command line, because the shell has to expand them and because they are facts about this build rather than about the shape of it. They add to what the profile says rather than replacing it, and a label named twice is one row carrying the higher layer's answer.

A matrix leg that needs one thing different does not need a profile of its own: PYTEST_HTML_REPORTER_LOGS=all in that job's environment overrides the profile for that job alone. And when a leg's report comes out wrong, pytest --report-profile=ci --report-show-config prints what it resolved and which layer decided each setting, into the pytest header — which is the part of a CI log people paste into issues. See pytest-html-reporter config for the same answer without running the suite.

Parallel runs with pytest-xdist

Runs distributed with pytest-xdist are gathered into a single report with no configuration at all — one build in Archives, one set of totals, one row per test, whichever way the tests were distributed.

$ pytest tests/ -n 2 --html-report=./report
$ pytest tests/ -n auto --dist loadfile --html-report=./report

How the workers hand their results back

1. Every worker collects the whole suite, and the plugin records each node id's collection index. A test therefore sits at the same position in every process.
2. Each worker builds ordinary record dicts as its tests finish. Records are made only of JSON-safe built-ins on purpose — that is what lets them cross a process boundary.
3. At the end of the session a worker writes its record list into config.workeroutput and renders nothing.
4. On the controller, each finished worker's output is read back and stored as it goes down.
5. The terminal-summary hook returns early on a worker. Only the controller reaches the render.
6. The report is built from records sorted by collection index and worker, so a parallel report reads exactly like a serial one whatever order the workers finished in.

Screenshots are written by whichever process ran the test — the workers share the controller's filesystem — into pytest_screenshots/ under the report base. An image is named <milliseconds>[-<worker id>]-<counter>: milliseconds alone collided because two tests can finish inside the same one, the counter settles that, and the worker id settles the cross-process case.

Two more consequences worth knowing. The controller never runs a test, so run-wide totals cannot be accumulated as tests go by; they are counted off the merged record list at the end. And an -n 8 run that is also one leg of a matrix writes its bundle once, from the controller, after it has merged all eight workers' records — never eight times with an eighth of the tests each.

$ pytest tests/ -n 8 --html-report=./report --report-shard=1/4

Reruns: how retries are represented

pytest-rerunfailures runs the whole setup/call/teardown protocol again for every retry, so a retried test arrives at the reporter once per attempt. The plugin keeps one record per test and uses the rerun field to say how many attempts it stands for. There is no rerun-specific flag: the plugin reacts to pytest-rerunfailures being installed rather than configuring it.

• Attempts are summed, and keptThe surviving record's rerun becomes the superseded record's count plus its own plus one, because the record being replaced is an attempt itself and either side may already stand for several. From 0.4.2 the record also carries the trail of what each of those attempts did.
• Screenshots and attachments survive a green retryThe surviving attempt back-fills them from the attempt it replaces when it has none of its own. A flaky test that captured the failing response on its first attempt should not lose it by then passing.
• Steps do notSteps are back-filled only when the surviving attempt recorded none. A retry that passed did so by running its own steps, and the failing attempt's tree beside a green test would describe a run that did not happen.
• The row keeps its placeThe folded record inherits the superseded record's collection index, so a retried test does not jump down the table.

Counting attempts is the only reliable signal available: --reruns, the ini key and @pytest.mark.flaky(reruns=n) can each set a different budget, and --only-rerun can stop the retries early, so no single number says how many attempts a given test will take.

The count then travels everywhere — into output.json as a per-test string and a per-suite total, into the JUnit <properties> as reruns and into each testcase's <system-out> as a reruns: N line, and into Analytics, where a non-zero rerun count is by itself enough to mark a test flaky. A retry inside a single build is the least ambiguous flake evidence there is: same code, same build, two different answers.

The attempt trail

One row carrying the outcome that stuck is the honest shape, but on its own it throws the interesting half away: the row shows the message of that attempt, and an attempt that stuck by passing has no message. A test that failed twice and then passed read PASS  2, and nothing anywhere in the report said what it had failed with — while two failures for two different reasons is a different bug report from the same failure twice.

Each attempt keeps four fields and not the record it came from: status, message, duration and worker. A shard bundle is the record list exactly as it stands in memory, so keeping whole attempts would multiply every bundle a matrix uploads by the number of times its flakiest tests were retried, and put a copy of each discarded attempt's logs and screenshots in it.

Under -n a worker sends back records it has already folded itself, so the controller keeps both sides' attempts and puts the record being replaced between them — it ran after its own attempts and before the one replacing it. The trail is parked outside the metrics table, so it is absent from the table's search index and from its CSV, Excel and print exports, and the Rerun column still exports the single number it always did.

A test that ran once has no trail and keeps the plain 0 the column has always shown, with the button disabled rather than only styled flat so it is not a tab stop either. So does a build archived before 0.4.2: the count was stored, the attempts behind it were not, and an empty panel would be worse than none.

Sharding a run across machines

A shard writes no report: no pytest_html_report.html, no output.json, no archive rotation. It writes one file — its record list exactly as it stands in memory, plus everything only that machine knows about itself — and the screenshots those records name.

The four shard flags

$ pytest tests/ --html-report=./report --report-shard=1/4
$ pytest tests/ --html-report=./report --report-shard=2/4
$ pytest tests/ --html-report=./report --report-shard=3/4
$ pytest tests/ --html-report=./report --report-shard=4/4

How a shard id becomes a directory

The shard is a directory under the report base, never a bare file beside the report. That is what makes merging in place safe: the merge clears <report base>/pytest_screenshots, while every source image sits under <report base>/shards/<id>/pytest_screenshots and is untouched.

Ids are sanitised into directory names. Everything outside A-Z a-z 0-9 . _ - becomes a dash, runs of dashes collapse, dots and dashes are stripped from both ends, and the result is capped at 64 characters.

The raw value is kept as the shard's label — 1/4 reads better than 1-4 in the report — while the sanitised value is the id, which names the directory and the screenshot folder.

Before a leg writes, it removes its own directory outright — never the whole report folder, which would take a sibling leg's screenshots with it, and never only pytest_screenshots, because a records.json left by a previous run of the same leg would survive and, if that run collected more tests, would be what the merge reads. If two differently-named legs sanitise to the same directory, the second write prints a warning naming both labels; it is never fatal, because this runs after every test in the leg has finished.

What a leg leaves on disk

report/
└── shards/
    ├── 1-4/
    │   ├── records.json
    │   └── pytest_screenshots/
    │       ├── 1788426639123-1.png
    │       └── 1788426639771-gw0-2.png
    ├── 2-4/
    │   ├── records.json
    │   └── pytest_screenshots/
    ├── 3-4/
    │   └── records.json
    └── 4-4/
        └── records.json

The bundle: records.json

One file per leg, written atomically — to a temporary file in the same directory, flushed, fsynced, then replaced over the top. A bundle is written from the terminal-summary hook, which is the last thing a CI leg does before the job may be cancelled or the runner reclaimed, so what is on disk is either the previous file or the whole new one, never half of one.

The file is identified by its schema string, not by its name: anything whose schema is not exactly pytest-html-reporter/records is walked past with a note, because a folder of CI artifacts is full of other JSON. A bundle whose version is higher than the reader understands stops the merge by name — silently dropping a quarter of a matrix produces a report that is wrong in a way nobody looking at it can see. Unknown keys survive a round trip untouched, at both the payload and the record level.

{
  "schema": "pytest-html-reporter/records",
  "version": 1,
  "generator": "pytest-html-reporter 0.4.3",
  "shard": { "id": "1-4", "label": "1/4", "assets": "pytest_screenshots" },
  "run": {
    "session_start": 1788426000.11,
    "session_end": 1788426724.53,
    "exitstatus": 1,
    "token": "github:1234567-2",
    "collected": 203,
    "hostname": "runner-1",
    "platform": "Linux 6.5.0-1024-azure",
    "python": "3.11.9",
    "pytest": "8.2.0",
    "plugins": ["cov-5.0.0", "html-reporter-0.4.3", "xdist-3.5.0"],
    "arguments": "tests/ -n 4 --html-report=./report --report-shard=1/4",
    "rootdir": "/home/runner/work/proj/proj",
    "environment": "staging",
    "build_info": [["Commit", "a1b2c3d"], ["Branch", "main"]],
    "xdist_workers": ["gw0", "gw1", "gw2", "gw3"]
  },
  "coverage": { "percent": 84.12, "statements": 5120, "covered": 4307 },
  "counts": { "records": 203, "collect": 1 },
  "records": [ { "nodeid": "tests/test_login.py::test_valid_user" } ]
}

The run block is the machine's own account of itself — host, platform, interpreter, pytest version, plugins, command line, rootdir, environment, build info, what it kept of each test's output — captured here because the merge often runs on a machine that ran none of the tests. counts is there for inspect and is never trusted by the merge, which counts the records it actually read.

Sequential legs on one machine

Unit, then integration, then e2e, on one machine, is the case where a fourth command to merge the three is a fourth thing to remember. The last leg passes --report-shard-merge, and three commands do the work of four.

$ pytest tests/unit        --html-report=./report --report-shard=unit --report-shard-reset
$ pytest tests/integration --html-report=./report --report-shard=integration
$ pytest tests/e2e         --html-report=./report --report-shard=e2e --report-shard-merge

<base>/shards is persistent, so it also holds whatever the last run left in it. A leg that was renamed or deleted between two CI runs leaves its bundle sitting there, and the next merging leg picks it up and reports tests that did not run — four tests run and the build says six, with nothing on the page saying where the other two came from. A clock cannot tell the two cases apart, because every bundle beside a merging leg was written before it, whether ten minutes ago by this run or yesterday by the last one. There are two answers, for two situations:

• A run token. A leg with a token merges only the bundles carrying the same one, and writes to stderr exactly which bundles it put aside, what token each carries, and the advice to clear the folder or use --report-shard-reset. It is taken from the CI system's own variables when you do not pass one — GITHUB_RUN_ID plus GITHUB_RUN_ATTEMPT on GitHub Actions, CI_PIPELINE_ID on GitLab, BUILD_TAG on Jenkins, CIRCLE_WORKFLOW_ID on CircleCI, BUILDKITE_BUILD_ID, BUILD_BUILDID on Azure Pipelines. A leg with no token merges everything, which is the only thing it can do: an empty token says nothing about which run a bundle came from.
• --report-shard-reset on the first leg, which empties <base>/shards before anything is written.

[pytest]
html_report = ./report
report_shard_run = nightly-42

A merging leg that cannot merge — a bundle from a newer release, a duplicate under an error policy — says so and carries on rather than raising. This runs after every test has finished, and a traceback there would leave the run with no report and no verdict. Its own bundle is on disk either way, and the message names the folder and tells you to run the merge command once it is sorted out.

Merging the shards

The merge is its own process, not another pytest run. A fifth pytest started in the report folder to do the merging would clean the screenshots on the way in and delete the images it was sent to collect. It is installed as a console script by setup.py and is also reachable as a module, which matters in containers that pip-installed a wheel, in tox environments and in a checkout somebody is trying the feature out in.

$ pytest-html-reporter merge ./artifacts --html-report ./report

$ python -m pytest_html_reporter merge ./artifacts --html-report ./report

It uses argparse and the standard library only. The package's install requirements are pytest and Pillow, and a merge command that needed a dependency of its own would be a command CI could not run.

Three subcommands

All three take the same positional PATH arguments — default, the current directory — and the same three shaping flags: --on-duplicate, --order and --strip-path-prefix. Those are shared deliberately, because they decide the answer rather than the output: inspect reporting a different set of tests from the merge that follows it would make it useless as a pre-flight check.

Paths may be directories, which are walked, or records.json files named directly, for a step that unpacked one artifact and knows where it put it. The walk is sorted at every level and duplicates are removed by absolute path. A folder of CI artifacts holds JUnit files, logs and a coverage report beside the bundles, and every one of them is walked past with a note rather than treated as a failure.

# one folder holding the downloaded artifacts
$ pytest-html-reporter merge ./artifacts --html-report ./report

# report and JUnit XML in one pass
$ pytest-html-reporter merge ./artifacts --html-report ./report --junit-xml ./report/junit.xml

# merging back into the folder the shards were written in - safe by design
$ pytest-html-reporter merge ./report --html-report ./report

# bundles named directly
$ pytest-html-reporter merge ./a/records.json ./b/records.json --html-report ./report

# XML only, no report
$ pytest-html-reporter junit ./artifacts -o ./junit.xml

# pre-flight: what is there, and what merging it would come to
$ pytest-html-reporter inspect ./artifacts
$ pytest-html-reporter inspect ./artifacts --json

The flags a pipeline actually reaches for

The complete list of every merge flag is in the CLI reference. These are the ones that change the shape of a CI job.

What the merge does, in order

1. Discover and read

   Every records.json under the given paths, in a stable order, without repeats. A bundle carrying no shard id is given its own directory's name as a fallback — hand-assembled bundles and artifacts unpacked by a step that renamed the folder both turn up without one.

2. Order the bundles

   Sorted by the natural key of the shard id, never by load order: runs of digits read as numbers, so 1-4, 2-4, 10-16 sort in that order. Two jobs downloading the same artifacts in different orders have to produce the same report, or the diff between two builds is unreadable. If two files claim one shard id, the one that finished later is kept and both paths are named in a note.

3. Normalise every record

   Every key is filled in from defaults and every type coerced, which is the whole defence against a bundle written by another release. Coercion is tolerant rather than strict: a duration that arrives as the string "1.2" or as null becomes 0.0 rather than losing the row. Then the node id is normalised, the suite name recomputed from it, and the record tagged with the leg it came from.

4. Deduplicate collectors, then tests

   On completely different grounds. Every process collects the whole suite, so a broken import is expected in all four bundles and is not a duplicate at all: one record is kept per collector, an ERROR supersedes a SKIP, and for the same status the longer message wins. Tests follow the duplicate policy.

5. Rebase every index

   Four shards each number their tests from zero. Collect rows are renumbered ahead of every test, test rows densely and uniquely, so the report's own sort can never fall back to a worker name that means nothing across machines. Under --order shard a suite split across shards stays contiguous instead of appearing twice.

6. Build the run metadata and the coverage

   One set of run facts assembled from every bundle's own account, never one machine's answer generalised to all of them; and coverage through five branches and no sixth.

7. Render, once

   Through the same render path a plain pytest run drives. A merged build assembled by different code is a build that drifts away from every other one on the report's own history.

Duplicate node ids

A node id that ran in more than one shard means the matrix overlaps, and somebody has to be told. There is no single right answer — a matrix that overlaps by accident wants merge, a matrix that retries a whole leg wants last, a gate that must not be fooled wants error — so it is asked rather than guessed. Every fold is reported on stderr and counted in the summary whichever policy is in force.

Node identity across machines

Two machines can run the same test under different roots — a Windows runner's backslashes, a container that checked the repo out at /src and a runner that used /home/runner/work/proj/proj. The same test under two spellings is two rows, two histories in Analytics and two entries in the JUnit file.

Node ids are normalised in a fixed order: backslashes become forward slashes, repeated slashes collapse, a leading ./ is stripped, then each --strip-path-prefix value is removed from the front. A prefix that would swallow the whole node id is not applied — the original is worth more than nothing. When the node id changes, the suite name is recomputed from it, so grouping follows identity instead of drifting away from it.

$ pytest-html-reporter merge ./artifacts \
    --html-report ./report \
    --strip-path-prefix /home/runner/work/proj/proj/ \
    --strip-path-prefix C:/actions-runner/_work/proj/proj/ \
    --strip-path-prefix /src/

A record whose node id is empty after all this is quarantined: dropped from the merge, counted in the summary and named in a note. It cannot be grouped, sorted against a duplicate, or linked to. Dropped rather than fatal, because the merge runs after every test in the matrix has finished, which is the worst moment there is to raise.

Screenshots across shards

The merge copies every PNG the merged records name into the report's own folder — by the names the records hold, never by globbing the shard's folder. A shard folder can still be holding images from a previous build of the same leg, and sweeping the folder would carry them into this report as tests that did not run in it. The merge owns <report base>/pytest_screenshots and removes it outright first: one build comes out of one merge.

Each shard's images land in a folder of their own, because the screenshot counter restarts at 1 in every process — two machines' first screenshots are both <ms>-1, and one would overwrite the other in a shared folder. An image a bundle named but did not carry is dropped rather than left pointing at nothing, counted as a missing screenshot, and named in a note that says which file the bundle promised.

report/
├── pytest_html_report.html
├── output.json
├── archive/
│   ├── output_1788425529.352927.json
│   └── output_1788426203.302967.json
├── pytest_screenshots/          <- owned by the merge, wiped and restaged
│   ├── 1-4/
│   │   └── 1788426639123-1.png
│   ├── 2-4/
│   │   └── 1788426639123-1.png   (same name, different shard, no collision)
│   └── 3-4/
└── shards/                      <- untouched when merging in place
    ├── 1-4/
    ├── 2-4/
    └── 3-4/

Provenance, notes and exit codes

Provenance is printed on every merge, with no threshold anywhere near it. There is no honest way to decide from inside a merge that a bundle is stale — every bundle beside a merging leg was written before it — so nothing is guessed and the times are shown. That is what turns yesterday's leg from a silent extra two tests in the totals into a line in the log of the run that merged it. Times are local, because this is read beside a CI log whose own timestamps are on the same clock.

Notes go to stderr and the summary goes to stdout, so a pipeline that captures one can discard the other, and inspect --json stays a clean document. -q suppresses the notes but never the summary and never an error.

merged 4 shards: 812 tests
  shard 1-4: 203 tests, finished 2026-09-03 11:42:07
  shard 2-4: 204 tests, finished 2026-09-03 11:44:31
  shard 3-4: 203 tests, finished 2026-09-03 11:41:55
  shard 4-4: 202 tests, finished 2026-09-03 11:45:02
  PASS 780, FAIL 21, SKIP 11
  3 collection records
  duplicates folded: 2
  missing screenshots: 1
  bundles superseded by a newer copy: 1
  ran on runner-1, runner-2, runner-3, runner-4 over 12 mins 04 secs
  report: /home/runner/work/proj/proj/report/pytest_html_report.html
  junit:  /home/runner/work/proj/proj/report/junit.xml

There are two independent failure flags, because they answer two different questions. --strict is about the merge being complete; --exit-code is about the tests. A pipeline that wants the build to go red on a failing test uses one; a pipeline that only wants to know all four artifacts arrived uses the other. The report is written either way — a merge that exits 1 has still produced the page that explains why.

inspect, as a pre-flight gate

inspect answers "did all the artifacts arrive, and do they overlap". It writes nothing and always exits 0 unless it could not start at all, so a pipeline that wants to fail loudly on a missing shard reads the count out of --json.

1-4                     203 records    1 collect  runner-1         ./artifacts/shard-1/records.json
2-4                     204 records    0 collect  runner-2         ./artifacts/shard-2/records.json
3-4                     203 records    1 collect  runner-3         ./artifacts/shard-3/records.json
4-4                     202 records    0 collect  runner-4         ./artifacts/shard-4/records.json

The JSON document carries the same information: one object per bundle, and a summary holding shards, tests, collects, statuses, reruns, duplicates_folded, quarantined, unreadable, superseded, unrecognised, the matrix's start, end and wall span, the distinct hosts, the agreed environment, any coverage notice, and every note the merge made in order.

$ pytest-html-reporter inspect ./artifacts --json > merge.json
$ test "$(python -c 'import json;print(json.load(open("merge.json"))["summary"]["shards"])')" = 4

A three-shard matrix, end to end

The shape of a sharded pipeline is always the same three moves: each leg writes a bundle, the bundles are uploaded as artifacts, one job downloads them all and merges once. Here it is with three legs, coverage, and the merged report published as an artifact.

Step one: the legs

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install pytest-html-reporter pytest-xdist pytest-cov
      - run: |
          pytest tests/ -n 4 \
            --cov=myapp --cov-report= \
            --html-report=./report \
            --report-shard=${{ matrix.shard }}/3
      # keep this leg's coverage data beside its bundle
      - if: always()
        run: cp .coverage report/shards/${{ matrix.shard }}-3/.coverage
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: shard-${{ matrix.shard }}
          path: report/shards/
          include-hidden-files: true

Three things about that job are worth saying out loud.

• -n 4 and --report-shard compose. Each leg is itself a four-worker xdist run, and writes exactly one bundle from its controller.
• The uploaded path is report/shards/, the whole directory. Uploading only the JSON leaves the merge with records naming pictures that are not there.
• if: always() on the copy and on the upload, so a leg with failing tests still contributes its half of the story: a red pytest step fails the step, and every later step without that condition is skipped — which would leave the shard with no coverage data and no artifact. And include-hidden-files, because .coverage is a dotfile and would otherwise be left out of the artifact.

Step two: what each leg uploads

artifacts/
├── shard-1/
│   └── 1-3/
│       ├── records.json
│       ├── .coverage
│       └── pytest_screenshots/
│           └── 1788426639123-1.png
├── shard-2/
│   └── 2-3/
│       ├── records.json
│       └── .coverage
└── shard-3/
    └── 3-3/
        ├── records.json
        └── .coverage

The nesting does not matter to the merge: the paths it is given are walked, so ./artifacts finds all three bundles wherever the download step put them. It matters to --coverage-data, which reads a named directory one level deep — every entry in it whose name starts with .coverage — and does not walk subdirectories. So the data files are gathered into one flat folder first.

Step three: the merge job

  merge:
    needs: test
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install pytest-html-reporter coverage
      - uses: actions/download-artifact@v4
        with:
          path: artifacts

      - name: every shard arrived
        run: |
          pytest-html-reporter inspect ./artifacts --json > merge.json
          test "$(python -c 'import json;print(json.load(open("merge.json"))["summary"]["shards"])')" = 3

      - name: gather the coverage data files
        run: |
          mkdir -p covdata
          i=0
          for f in $(find artifacts -name '.coverage'); do
            i=$((i + 1))
            cp "$f" "covdata/.coverage.$i"
          done

      - name: merge
        run: |
          pytest-html-reporter merge ./artifacts \
            --html-report ./report \
            --junit-xml ./report/junit.xml \
            --coverage-data ./covdata \
            --coverage-target 80 \
            --title "NIGHTLY" \
            --environment staging \
            --build-info "Commit=${{ github.sha }}" \
            --build-info "Branch=${{ github.ref_name }}" \
            --strict

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: report
          path: report/

What comes out is one build: one pytest_html_report.html, one output.json, one archived build, one trend point, one junit.xml with exactly one <testcase> per node id, and one coverage figure combined from the three legs' data.

Every row in that panel is something a shard reported about itself, never the merging machine's answer — the merging machine ran no tests. Where the legs disagree the panel says so rather than picking: an Environment the shards do not agree on is left blank with a note listing the values, and captured-output settings that differ produce one row per leg instead of one summarised row that would be a lie. The run header shows the span of the matrix rather than the sum of the legs, so three shards that each took ten minutes in parallel are reported as ten minutes.

JUnit XML

There is no JUnit input path anywhere in this plugin. Nothing reads a JUnit XML — not the plugin, not the merge command. The writer is a pure function over the record dicts, which is what makes a live pytest run and a merge produce the same document from the same input.

Why it is built from records rather than merged from XMLs

Every merging tool available gets one of four things wrong, and building from records answers all four at once.

• Counts are summed. Tools that merge XMLs add up the tests and failures attributes they find, so a test that ran on two machines is counted twice while only one element survives — the header and the body of the same file then disagree. Here every count is recomputed from the elements actually emitted.
• A duplicate <testcase> is read differently by every consumer. GitLab keeps the first and Jenkins keeps the last, so a flaky test that failed and then passed is reported red by one and green by the other. Here there is exactly one element per node id; two records that mangle to one address are renamed name [2], with a warning, rather than emitted twice.
• Azure DevOps derives the end of the run from the timestamp plus the sum of the per-test times. So a per-test time that is short by a teardown, or rounded to 0.0, makes a forty-minute matrix read as instantaneous. Here testcase@time is the sum of the phase milliseconds, not the two-decimal duration the table shows.
• Properties are invisible to two of the four consumers. GitLab ignores them outright and Jenkins reads them only with keepProperties on, so which shard ran a test and how many retries it took are written into <system-out> as well as into <properties>.

It is one <testsuite>, not one per shard: Azure's run-level timing across several suites is ambiguous and every other consumer flattens them anyway, so shard identity lives in the properties and in each testcase's captured output instead. Writing is atomic — a temporary file beside the target, then a rename — because a CI collector very often watches the directory it will read from, and half a document is a parse error attributed to the tests rather than to the reporting.

Writing it, from a run or from a merge

$ pytest tests/ --html-report=./report --report-junit=./report/junit.xml
$ pytest tests/ --html-report=./report --report-junit=./report/junit.xml --report-junit-xpass=fail

$ pytest-html-reporter merge ./artifacts \
    --html-report ./report \
    --junit-xml ./report/junit.xml \
    --junit-logging failed \
    --junit-suite-name e2e

$ pytest-html-reporter junit ./artifacts -o ./junit.xml --no-junit-attachments

<?xml version="1.0" encoding="utf-8"?>
<testsuites name="pytest tests" tests="3" failures="1" errors="1" skipped="0" time="724.000">
  <testsuite name="pytest" tests="3" failures="1" errors="1" skipped="0" time="724.000"
             timestamp="2026-09-03T13:00:00" hostname="merged">
    <properties>
      <property name="pytest-html-reporter" value="0.4.3" />
      <property name="shards" value="4" />
      <property name="shard.ids" value="1-4,2-4,3-4,4-4" />
      <property name="shard.hosts" value="runner-1,runner-2,runner-3,runner-4" />
      <property name="reruns" value="2" />
      <property name="duplicates-folded" value="0" />
    </properties>
    <testcase classname="tests.test_login" name="test_valid_user" time="1.204">
      <properties>
        <property name="owner" value="payments-team" />
        <property name="jira" value="PROJ-123" />
        <property name="jira" value="PROJ-987" />
        <property name="severity" value="blocker" />
      </properties>
      <system-out>[[ATTACHMENT|pytest_screenshots/1-4/1788426639123-1.png]]
shard: 1/4 (runner-1)
reruns: 2</system-out>
    </testcase>
    <testcase classname="tests.test_cart" name="test_add[qty=2]" time="0.400">
      <failure message="AssertionError: assert 1 == 2">AssertionError: assert 1 == 2</failure>
      <system-out>shard: 2/4 (runner-2)</system-out>
    </testcase>
    <testcase classname="tests.test_api" name="(collection error)" time="0.000">
      <error message="collection failure">ImportError: No module named 'requests'</error>
      <system-out>shard: 3/4 (runner-3)</system-out>
    </testcase>
  </testsuite>
</testsuites>

The order inside <system-out> is fixed: attachments first, because a collector scans for them; then the shard and the rerun count, the two facts that would otherwise only live in properties nobody reads; then the captured output, which is the longest and least machine-read part. There is at most one <system-out> and one <system-err> per testcase, because a document with two of them is one whose second some parsers drop and others concatenate.

Traceability ids on the testcase

This is the machine-readable half of traceability. Xray, Zephyr and TestRail all ingest a test's issue key from a testcase property, and none of them opens an html report, so an id that only ever reaches a badge is invisible to exactly the tools that were supposed to consume it.

• The property name is the marker name. A suite that has to emit test_key for Xray writes @pytest.mark.test_key("ACME-42") and gets test_key, rather than a vocabulary this plugin chose.
• Only owner, severity and the markers you named are written. A document that started listing every user marker would change what a collector reports for suites that asked for nothing.
• A marker with empty brackets writes no property. An empty one is a row every consumer displays and none can use.
• Two jira markers are two properties. Every traceability marker is a set — a test that closes two tickets belongs in both.
• Severity is written once, already resolved. It is the one exception to that, because a testcase carrying both normal and critical is one no consumer can rank; the report has already picked between them by the time the XML is written.

owner and severity need no configuration, so they are written by the merge and junit subcommands as well. Pattern markers need to be named, and only merge takes --report-link-pattern — it reads them off its own argv, having no ini file to ask. See markers that link for the html side of the same configuration.

How each status becomes an element

The mapping is explicit and closed on purpose. A canonical CI file is the last place that should be guessing, so a status this version has never heard of is written as an <error> and warned about loudly rather than absorbed into a plausible total.

Test coverage

Coverage is read, never re-measured. Nothing in that path is allowed to take a test run down: every branch either returns data or returns nothing, and a failure to read is reported into the page rather than raised. By the time it runs the tests are over, and failing the run over a decoration would turn a green build red.

Where the percentage comes from

On a live run the sources are tried in this order, stopping at the first that yields data.

1. --report-coverage none — nothing at all, and no tab.
2. --report-coverage-file PATH — read that file. If nothing there can be read as coverage data, the tab shows a notice naming the path. It does not fall through to the other sources, because you said where to look.
3. The live pytest-cov run. The coverage object is asked for its own JSON report through coverage.py's public API, so the tab and your terminal always agree. This is read rather than raced for: pytest-cov stops, saves and combines its data — including every xdist worker's share — long before the report is written. The tab says Measured by pytest-cov during this run.
4. Discovery — coverage.json, then coverage.xml, looked for beside the report, at the rootdir and in the working directory, first hit wins. Only those two deliberate report artifacts are looked for: a stray .coverage data file is often left over from a run days ago, and quietly reporting yesterday's number is worse than reporting none.
5. If pytest-cov ran but measured nothing, the tab explains why rather than showing a setup guide to somebody who has already installed the plugin and passed the flag.

The format is sniffed from the first byte, not the extension — < is Cobertura XML, { is coverage.py JSON, anything else is tried as a .coverage SQLite data file. CI hands these files whatever name the pipeline felt like.

# the ordinary case: pytest-cov measures, the reporter reads
$ pytest tests/ --cov=myapp --cov-report=html --cov-fail-under=80 --html-report=./report

# read an artifact another step produced
$ pytest tests/ --html-report=./report --report-coverage-file=./coverage.xml

# no Coverage tab at all
$ pytest tests/ --html-report=./report --report-coverage=none

The per-file split

Files are listed worst first, by percentage then by missing lines then by name. That is both the useful default and the only defensible way to cut the list: a cap that kept the alphabetical head would hide exactly the files the tab is opened to find. The full count is always recorded, so the cap note can say how many were left out and how to see them all.

Each row carries the file's percentage, statements, covered, missing, excluded, branches, partial branches and the missing line numbers written the way coverage.py writes them — 12-15, 88, 91-140 — trimmed at a comma and never mid-range, because 91-14 reads as a line range that does not exist. Paths are relativised against the rootdir, since four directories of CI checkout in front of src/api.py identify nothing. A file with no branches shows an em dash rather than 0/0: having no branches to cover is not the same statement as covering none of them, and the Branches and Partial tiles appear only when branch coverage was actually switched on.

Colour, targets and drift

The ring is green at 90% and above, amber at 75%, red below — unless the project has stated its own bar with --cov-fail-under (or --coverage-target on a merge), in which case that is the line the colour is drawn at and the tab says so. A report should not disagree with the build that just passed or failed beside it, so a hair of slack keeps 79.999999999 against a target of 80 from reading as a failure in the report while the build itself passed.

The percentage is written into output.json alongside the test counts, so it travels with the archived builds. That is what gives the tab its +0.8 since the last build chip and its trend line. The delta compares this build against the most recent earlier build that measured any coverage, skipping builds that ran without it rather than counting them as a drop to zero; a movement under 0.05 reads as no change. A build that ran without coverage leaves a gap in the line rather than a drop to zero, because the coverage key is simply absent from that build's output.json — and absent means "not measured", which is a different answer from zero and the true one.

The annotated source

The one thing a summary cannot replace is the source, line by line, with the missed lines marked. Generate it with --cov-report=html and the tab links to it.

It is linked, never embedded. An iframe would break the property the whole reporter is built on — one file you can mail, publish as a CI artifact or open off a stick — and it would break silently, showing an empty frame wherever the folder did not travel. The link is offered only when the index file exists and was written after this run started: coverage.py names that folder whether or not the HTML report was asked for, so a stale htmlcov sits there looking exactly like a fresh one, and annotations that disagree with the summary beside them are worse than no link.

Coverage on a merged build

The merge has five branches, in this order, and no sixth.

1. --report-coverage none — nothing. Reachable only through a --report-shard-merge leg, which reads the pytest flag; the merge command has no such flag, so coverage there is auto unless you name a source.
2. --report-coverage-file — read it. If nothing there can be read, the build gets no coverage and a notice naming the path.
3. --coverage-data — collect every named file, and every entry named .coverage* directly inside every named directory, and combine them with the coverage package. The source line reads N combined data files. A missing coverage package, no files matched, or a combine that raised are each reported as a notice rather than a failure.
4. Exactly one bundle measured coverage — use it, with a notice saying it covers that shard's share of the run.
5. More than one measured, or none did — no coverage, and a notice telling you what to do about it.

The data files are copied into a temporary directory before being combined, because coverage.py's own combine() deletes the files it reads — and these are somebody's CI artifacts, which are very often the only copy.

# let the merge combine the data files itself
$ pytest-html-reporter merge ./artifacts \
    --html-report ./report \
    --coverage-data ./covdata \
    --coverage-target 80

# or combine first with coverage.py and hand over one report
$ coverage combine ./covdata/.coverage.*
$ coverage json -o ./coverage.json
$ pytest-html-reporter merge ./artifacts \
    --html-report ./report \
    --report-coverage-file ./coverage.json \
    --coverage-target 80

Troubleshooting: the Coverage tab is empty

Work down this list. The first one that applies is the answer.

Whichever source the numbers do come from, the tab states it — Measured by pytest-cov during this run, or Read from coverage.xml, written 2026-08-31 20:23 — so you can always tell which of these you are in.

CI recipes

Five things are worth getting right whatever the platform.

• Upload the whole shard directoryreport/shards/<id>/, bundle and screenshots together — not only records.json.
• Ask the merge for the XML, not the legs--report-junit on a plain shard is ignored with a notice, and a glob that collected one XML per leg plus the merged one would count every test twice.
• Run inspect before mergeIt writes nothing and exits 0 whatever it finds, so a pipeline that has to fail loudly on an artifact that did not arrive reads the shard count out of inspect --json.
• Pick your failure flag deliberately--strict catches the quiet failures — a folded duplicate, a missing image, a bundle superseded by a second copy. --exit-code catches failing tests. They are independent.
• On separate machines, --html-report only decides where each leg's own shards directory goesWhat has to match is the artifact you upload, not the path.

GitHub Actions

The full three-shard workflow is the worked example above. For a single unsharded job there is a dedicated composite Action that installs Python, runs pytest with the plugin, and publishes the result to the workflow job summary, a self-editing pull request comment, a downloadable artifact and GitHub Pages — with step outputs a later step can gate on.

- uses: actions/checkout@v4
- uses: prashanth-sams/pytest-html-reporter-action@v1

GitLab CI

GitLab's parallel keyword hands each job a CI_NODE_INDEX and a CI_NODE_TOTAL, which is exactly a shard id. The run token is derived automatically from CI_PIPELINE_ID, and the merged XML goes into reports:junit: so the pipeline's own test tab reads it — one entry per test, because the document has exactly one <testcase> per node id.

test:
  parallel: 4
  script:
    - pip install pytest-html-reporter
    - pytest tests/ --html-report=./report --report-shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  artifacts:
    when: always
    paths: [report/shards/]

merge:
  needs: [test]
  script:
    - pip install pytest-html-reporter
    - pytest-html-reporter merge ./report
        --html-report ./report
        --junit-xml ./junit.xml
        --exit-code
  artifacts:
    when: always
    paths: [report/]
    reports:
      junit: junit.xml

The merge reads ./report and writes ./report, which is safe by design: the shards live in a subtree the merge never clears, and only report/pytest_screenshots is wiped and restaged.

Jenkins

Jenkins is the sequential case — several stages on one agent, in one workspace — so it is the one that --report-shard-merge was written for. The run token is derived from BUILD_TAG, which is jenkins-${JOB_NAME}-${BUILD_NUMBER} and so is unique across jobs, unlike the bare build number, which only counts within one. The first stage passes --report-shard-reset so a stage that was renamed or removed since the last build cannot leave a bundle behind for this build to merge.

# stage: unit
$ pytest tests/unit --html-report=./report --report-shard=unit --report-shard-reset

# stage: integration
$ pytest tests/integration --html-report=./report --report-shard=integration

# stage: e2e - the last leg merges the three and writes the one report and the one xml
$ pytest tests/e2e --html-report=./report \
    --report-shard=e2e --report-shard-merge \
    --report-junit=./report/junit.xml \
    --archive-count=60

Publish the results from the pipeline's post block with Jenkins' own steps: junit 'report/junit.xml' for the test result page, and archiveArtifacts artifacts: 'report/**' to keep the HTML report against the build. Because the workspace persists between builds, report/output.json and report/archive/ survive, which is what gives this job a real trend chart and a populated Analytics tab rather than a one-build view.

Keeping the report file down

Set no retention at all and every build is kept for ever, which is what eventually makes a report slow to open: a retained build costs roughly 5KB of the page, so an hourly job reaches a multi-megabyte report inside a couple of months.

Retention: three limits that intersect

An archived build has to satisfy every limit you set in order to survive. Retention runs after the trend has been drawn and before the Archives section and Analytics are built, so the history Analytics reads is exactly the history the report ships with.

When both a days limit and a since date are given, the stricter of the two wins, so neither can widen the other. All three are validated by name: a non-numeric or negative count, a non-numeric or negative days value, or an unparseable date each fail the run rather than being ignored — and on the merge command they are checked before anything is written, so a typo cannot abort a build that had already rotated the archive.

# a nightly job: keep a month of history, never more than 60 builds
$ pytest tests/ --html-report=./report --archive-days=30 --archive-count=60

# the same limits applied by a merge
$ pytest-html-reporter merge ./artifacts --html-report ./report \
    --archive-days 30 --archive-count 60

# keep nothing but this build
$ pytest tests/ --html-report=./report --archive-count=0

Retention is a property of the job rather than of one run, so the ini file is usually the better place for it: set it once and every invocation, however it is started, keeps the same window.

[pytest]
html_report = ./report
archive_count = 60
archive_days = 30

The per-test evidence caps

The other half of the size question is how much each test carries. Four settings decide it, and all four have an all / failed / none narrowing plus a numeric cap where a cap makes sense.

$ pytest tests/ --html-report=./report \
    --report-logs=failed --report-log-limit=5000 \
    --report-attachments=failed \
    --report-steps=failed --report-step-limit=100 \
    --archive-days=30 --archive-count=60