GitHub Action
One step installs the plugin, runs pytest, and delivers the result to four places at once — the workflow job summary, a pull request comment that edits itself on every push, a downloadable artifact, and a set of step outputs a later step can act on. This is the reference for every input, every output, and every workflow shipped in the repository.
What the action does
pytest-html-reporter-action is a composite GitHub Action that wraps the pytest-html-reporter plugin. One step optionally installs Python and your dependencies, runs pytest with the plugin’s --html-report flag, and then publishes the result.
- In the job summaryThe headline, the counts, every failure with its message, the suites ranked by damage, the slowest tests, and coverage when the run measured any. No clicking through to an artifact to find out what broke.
- On the pull requestThe same summary as one comment that is updated on every run rather than a new comment each time.
- As an artifactThe full interactive HTML report — Dashboard with its Trends chart, Analytics, Test Suites, Test Metrics, Test Steps, Archives, Screenshots, API Logs and Test Coverage. See the report tour.
-
As step outputs
passed,failed,pass-rate,coverage,statusand a dozen more, so a later step can post to Slack, open an issue, or gate a deploy.
On top of the plugin it adds the machinery a CI run needs: it resolves the report path up front so strftime placeholders cannot be expanded twice, probes pytest --help so pinning an older plugin degrades rather than aborting the run, carries the build archive between runs with actions/cache so Trends and Archives have history to read, gates the job on pass rate and coverage, and escapes every test name on the way into the summary — because on a fork’s pull request that text belongs to somebody else.
prashanth-sams/pytest-html-reporter-action. The -action suffix exists only because the plugin already owns the shorter name, and it appears in the uses: line and nowhere else.Quick start
This is the whole of a first setup: install, run, report, upload, fail if anything failed. Everything else on this page is an addition to this shape.
# The whole thing: install, run, report, upload, fail if anything failed.
name: tests
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: prashanth-sams/pytest-html-reporter-action@v1
with:
python-version: '3.12'
requirements: requirements.txt
tests: tests/
title: CI
Nothing else is required. Every input is a string and every input has a default, so you set as few or as many as you like — and booleans are the strings 'true' and 'false', because a workflow input is always a string.
The uses line, and which ref to pin
The project follows semantic versioning. The v1 tag moves to the newest v1.x.y release, so @v1 picks up fixes; pin an exact release to hold still.
- uses: prashanth-sams/pytest-html-reporter-action@v1
with:
tests: tests/
- uses: prashanth-sams/pytest-html-reporter-action@v1.0.0
with:
tests: tests/
What the summary says
The same markdown is used for the job summary, the pull request comment and the summary output. It is built from the run’s output.json, in this order.
- Heading —
## <icon> <summary-title>, where the icon is a tick forPASS, a cross forFAILand a question mark forUNKNOWN. - Headline — one middot-separated line: the status, then N tests across M suites (with in 1m 14s when a wall clock was measured), then the pass rate when one could be computed, then the rerun count when there were any.
- Counts table — Passed and Failed always; Error, Skipped, xPassed, xFailed and Rerun only when non-zero.
- Coverage line — when the run measured coverage: the percentage, whether it is branch or line, and how many statements of how many were covered.
- Failures — one collapsed block per failed or errored test, in report order, up to
failure-limit. Each issuite › test, with the captured message in a fence trimmed to 1,200 characters. Anything beyond the limit is counted in a trailing line. - Suites — a table sorted by failures plus errors, capped at
suite-limit. Rendered only when the run had more than one suite. - Slowest tests — a collapsed table, up to
slowest-limit. Skipped tests are excluded, because a skipped test’s duration is the cost of deciding to skip it, and so are zero-duration tests. - Links — Open the report (from
report-url), Download the artifact, and the workflow run.
When no report was written at all, the heading is followed by a single paragraph naming output.json and the directory it was expected in, and nothing else.
<details> first, so nothing after the cut is silently folded away.Inputs
Every input is a string, every input has a default, and no input is required. Nothing is interpolated into a shell: every input reaches bash as a PHR_* environment variable, never as ${{ ... }} spliced into a run: body — that would make pytest-args shell source.
What to run
| Input | Type | Required | Default | What it does |
|---|---|---|---|---|
tests | string (multiline) | no | '' | Test paths to hand to pytest, one per line. Blank lines are dropped. Leave empty to let pytest pick its own targets from your rootdir configuration. |
pytest-args | string | no | '' | Extra arguments appended to the pytest command, parsed the way a shell would parse them — quotes respected, via shlex.split. Unbalanced quoting fails the run rather than being guessed at. |
working-directory | string | no | . | Directory to run pytest in. Every path is resolved against it. The action fails up front if it does not exist. |
The environment
The install step is skipped entirely only when install is false and both requirements and extra-packages are empty.
| Input | Type | Required | Default | What it does |
|---|---|---|---|---|
python-version | string | no | '' | Python to set up with actions/setup-python@v5 before running. Leave empty to use whatever Python is already on the runner. |
install | boolean string | no | 'true' | Install pytest-html-reporter before running. Set to 'false' when your own requirements file already pins it. |
plugin-version | string | no | '' | Version specifier for the plugin. A value starting with a digit becomes ==<value>; anything else is appended raw, so >=0.3.8 works. Empty installs the latest release. Ignored when install is false. |
requirements | string | no | '' | Requirements file to install before running. Resolved against working-directory, and installed before the plugin. |
extra-packages | string | no | '' | Further pip packages to install, whitespace separated. Globbing is disabled while installing, so a name containing * is not expanded against the working directory. |
The report
Each of these maps onto a plugin flag, and each is left out of the command entirely when empty — so the plugin’s own defaults, and any report_* key in your pytest.ini, still apply. Setting one here overrides the ini key. Every flag is also checked against pytest --help before it is passed; see flag probing.
--html-report is the report-path input, and --report-link is the report-links input.| Input | Type | Required | Default | What it does |
|---|---|---|---|---|
report-path | string | no | report | Where the report goes, passed as --html-report. A value containing .html anywhere names the file; anything else names a directory. strftime placeholders are expanded once, before the run starts. Defaults to report rather than the plugin’s own default of the working directory, because a CI job wants the report, its output.json and its archive/ in one uploadable folder. |
title | string | no | '' | Report title, passed as --title. Shown cut to 20 characters, with the full text as the heading’s tooltip. |
environment | string | no | '' | Name of the environment under test, passed as --environment. |
build-info | string (multiline) | no | '' | Extra KEY=VALUE details shown in the report, one per line, each passed as its own --build-info. Blank lines and lines starting with # are ignored. |
report-links | string (multiline) | no | '' | Extra side-nav links, one LABEL=URL per line, each passed as its own --report-link. A relative path works; a link with a scheme the plugin does not know is dropped silently. |
archive-count | integer string | no | '' | Maximum builds kept in the Archives section (--archive-count). Empty keeps every one. '0' deletes the whole archive and hides the Archives section; '1' is read as “this build and no others”. |
archive-days | number string | no | '' | Keep only builds from the last N days (--archive-days). Accepts fractions, e.g. '0.5'. |
archive-since | string | no | '' | Delete archived builds older than this date (--archive-since), given as YYYY-MM-DD or 'YYYY-MM-DD HH:MM'. Read in the runner’s timezone, which is UTC. |
report-logs | enum: all, failed, none | no | '' | Whose captured stdout, stderr and logging to keep. Validated up front. |
report-log-limit | integer string | no | '' | Maximum characters of captured output kept per test; '0' keeps everything. Must be a non-negative whole number. |
report-attachments | enum: all, failed, none | no | '' | Whose attachments to keep. |
report-attachment-limit | integer string | no | '' | Maximum characters kept per attached payload; '0' keeps everything. |
report-steps | enum: all, failed, none | no | '' | Whose test steps to keep. |
report-step-limit | integer string | no | '' | Maximum steps kept per test; '0' keeps every one. |
report-coverage | enum: auto, none | no | '' | Whether to build the Test Coverage tab. |
report-coverage-file | string | no | '' | Read coverage from this file — a coverage.json, a Cobertura coverage.xml or a .coverage data file — instead of looking for one. Naming a file here is final: if it cannot be read the Coverage tab is blank and the coverage this run measured is not used instead. The action warns when that happens. |
report-coverage-limit | integer string | no | '' | Maximum files listed on the Coverage tab; '0' lists every one. |
report-open | enum: auto, always, none | no | '' (see below) | Whether to open the finished report in a browser. Set it only on a self-hosted runner with a desktop. |
--report-open whether or not you set it, and none when you have not. The plugin’s default is to open the finished report and it declines on a runner — unless a pytest.ini says report_open = always, which skips every check and hands the report to a console browser with no TTY, CI or DISPLAY test at all. auto is not a defence the action controls, because the CLI beats the ini.0, and 0 means “no limit” for every one of the *-limit inputs — so '-1' would silently mean the exact opposite of what it looks like. The action fails up front and says so.Publishing
These decide where the result goes: the job summary page, a pull request comment, a workflow artifact, a Pages artifact, and the cached build history.
| Input | Type | Required | Default | What it does |
|---|---|---|---|---|
job-summary | boolean string | no | 'true' | Write the run’s summary to the workflow’s job summary page. When there is no $GITHUB_STEP_SUMMARY to write to, the markdown is printed to stdout instead. |
summary-title | string | no | pytest-html-reporter | Heading used for the job summary and the pull request comment. |
failure-limit | integer string | no | '10' | Maximum failures listed in the summary and the comment; the rest are counted in a trailing line. '0' lists none. A non-numeric value falls back to the default rather than abandoning the run. |
suite-limit | integer string | no | '20' | Maximum suites listed. The Suites table is only rendered at all when the run has more than one suite. |
slowest-limit | integer string | no | '5' | How many of the slowest tests to list; '0' lists none. |
report-url | string | no | '' | Link to the published report, shown in the summary and the comment as Open the report — for instance your GitHub Pages URL. |
comment | boolean string | no | 'false' | Post the summary as a sticky comment on the pull request, updating the same comment on every run instead of piling new ones up. |
comment-mode | enum: always, on-failure | no | always | When to comment. on-failure compares the report’s status against PASS, so a run that produced no report (UNKNOWN) also comments. |
pr-number | integer string | no | '' | Pull request to comment on. Only needed when the event does not carry one — a workflow_run job, say. Empty uses the event’s own number. Must be digits. |
github-token | string | no | ${{ github.token }} | Token used to post the comment. Needs pull-requests: write. A pull_request run from a fork gets a read-only token and cannot comment. An empty value makes the comment step warn and return rather than error. |
upload-artifact | boolean string | no | 'true' | Upload the report directory as a workflow artifact with actions/upload-artifact@v4. |
artifact-name | string | no | pytest-html-report | Name of the uploaded artifact. Must be unique within a run — upload-artifact@v4 refuses two artifacts of the same name. |
artifact-retention-days | integer string | no | '' | Days to keep the artifact. Empty uses the repository default. |
artifact-include-history | boolean string | no | 'true' | Include the archive/ folder of past builds in the artifact. Set to 'false' to upload only the current report; the action then adds an exclusion pattern for <report-dir>/archive. |
artifact-if-no-files-found | enum: warn, error, ignore | no | warn | What upload-artifact does when nothing matched. |
artifact-overwrite | boolean string | no | 'false' | Replace an artifact of the same name instead of failing. A matrix is better served by giving each job its own artifact-name. |
pages-artifact | boolean string | no | 'false' | Also upload the report as a GitHub Pages artifact with actions/upload-pages-artifact@v3, ready for a following job to deploy with actions/deploy-pages. |
history | boolean string | no | 'false' | Carry the archive/ folder and output.json between runs with actions/cache, so the Trends, Archives and Analytics tabs have more than one build to show. |
history-key | string | no | pytest-html-reporter-history | Cache key prefix used for the build history. The full key appends runner.os, github.ref_name, github.run_id and github.run_attempt. |
Gates
Four inputs decide whether the job goes red. They are evaluated together, and every reason that applies is collected — a failing run tells you everything that was wrong, not just the first thing. See thresholds and gates.
| Input | Type | Required | Default | What it does |
|---|---|---|---|---|
fail-on-error | boolean string | no | 'true' | Fail the job when pytest fails — any non-zero exit code other than 5, which fail-on-empty owns. It also covers the case where pytest exits 0 but wrote no output.json. Set to 'false' to always go green and decide with the outputs yourself. |
fail-on-empty | boolean string | no | 'true' | Fail the job when pytest collects no tests at all (exit code 5). Left on, because a mistyped -k that quietly produces a green run is worse than a red one. When exit code 5 occurs this input alone decides; fail-on-error is not consulted. |
min-pass-rate | number string | no | '' | Fail the job when the pass rate falls below this percentage. Empty applies no threshold. A run where nothing decisive happened fails rather than passing by default. A non-numeric value is warned about and ignored. |
min-coverage | number string | no | '' | Fail the job when coverage falls below this percentage. Empty applies no threshold. A run that produced no coverage data fails rather than passing by default. |
Outputs
Every output is declared in action.yml and wired to one of the action’s internal steps. Read them from a later step by giving the action an id.
| Output | Type | What it holds |
|---|---|---|
status | string | PASS, FAIL, or UNKNOWN when no report was produced. Taken from the report’s own status field when it is PASS or FAIL, otherwise derived: FAIL if there were failures or errors, PASS otherwise. |
total | integer | Tests executed — passes, failures, errors, skips, xpasses and xfails summed. Reruns are attempts, not tests, and are not counted here. |
passed | integer | Tests that passed. |
failed | integer | Tests that failed. |
errors | integer | Tests that errored. |
skipped | integer | Tests that were skipped. |
xpassed | integer | Tests that passed while expected to fail. |
xfailed | integer | Tests that failed as expected. |
rerun | integer | Rerun attempts recorded across the run. |
suites | integer | Test suites in the run. |
pass-rate | number string | passed / (passed + failed + errors) as a percentage, trimmed of trailing zeros. Empty when nothing decisive ran. |
coverage | number string | Coverage percentage, or empty when the run produced none. |
tests-duration | number | Summed test durations in seconds, to 2 places. This is what the report records — the run’s own wall clock is not written to output.json — so it leaves out collection, session fixtures and the gaps between tests. |
wall-clock | number string | Seconds the pytest step took, measured by this action with date +%s either side of the run. |
exit-code | integer | The exit code pytest returned. See the table below. |
report-file | path | Absolute path of the generated HTML report. |
report-dir | path | Absolute path of the directory holding the report. |
json-path | path | Absolute path of the run’s output.json. |
summary | markdown | The markdown summary, for reuse in a later step. Multi-line, so pass it through env: rather than splicing it into a command. |
artifact-id | string | Id of the uploaded artifact, when one was uploaded. |
artifact-url | url | URL of the uploaded artifact, when one was uploaded. Rendered in the summary as Download the artifact. |
Reading the outputs
Pass multi-line values such as summary through env: rather than splicing them into a command — the summary is markdown built partly out of test names, and a test name is not something to hand to a shell.
- uses: prashanth-sams/pytest-html-reporter-action@v1
id: report
with:
tests: tests/
fail-on-error: 'false' # decide for yourself, below
- name: Tell the team
if: steps.report.outputs.status != 'PASS'
run: gh issue create --title "Nightly run: $FAILED failing" --body "$SUMMARY"
env:
GH_TOKEN: ${{ github.token }}
FAILED: ${{ steps.report.outputs.failed }}
# Through env, never spliced into the command: the summary is multi-line
# markdown built partly out of test names, and a test name is not
# something to hand to a shell.
SUMMARY: ${{ steps.report.outputs.summary }}
if: always() to run at all. The tidier way round is fail-on-error: 'false' plus your own decision — then nothing has failed and the outputs are simply there.Pull request comments
Reach for this when you want reviewers to see the result without opening the run. This is the same-repository case: the job needs pull-requests: write, which a pull request from a fork will not have. For those, see fork pull requests.
# A sticky summary on every pull request, edited in place on each push.
#
# This is the same-repository case. A pull request from a fork gets a
# read-only token and cannot comment - see fork-pr.yml for that.
name: tests
on:
pull_request:
permissions:
contents: read
pull-requests: write
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: prashanth-sams/pytest-html-reporter-action@v1
with:
tests: tests/
comment: 'true'
comment-mode: always
environment: ci
build-info: |
branch=${{ github.head_ref }}
commit=${{ github.sha }}
comment: 'true' posts the summary as a sticky comment: the body carries the hidden marker <!-- pytest-html-reporter-action -->, and the action paginates the pull request’s existing comments looking for it. If it finds one it updates that comment; otherwise it creates one. So the next push edits the comment rather than adding another.
When the comment step runs
All three conditions must hold:
commentis'true', and- the event is
pull_requestorpull_request_target, orpr-numberis set explicitly, and comment-modeisalways, or the run’sstatusis notPASS.
That second clause is why pr-number exists: a workflow_run job carries no pull request of its own.
It warns, it does not fail
Every failure mode here produces a warning annotation and lets the run continue, because a report is not worth turning a run red over: an empty github-token, no comment body produced, no pull request in the event, or an API call that threw. Look for a warning annotation rather than a red step.
Fork pull requests
A pull_request run triggered from a fork is handed a read-only GITHUB_TOKEN by GitHub, on purpose: the code being tested is not yours. The action does not fight this — it catches the API error, warns, and carries on. Commenting on a fork’s pull request takes two workflows.
Reach for this when you accept pull requests from forks and want them commented on. The first workflow runs the tests with a read-only token and writes the summary and the claimed pull request number out as a small artifact. The second, triggered by workflow_run, runs from your default branch with a writable token and none of the fork’s code, and posts the comment.
# Commenting on a pull request that came from a fork.
#
# A `pull_request` run from a fork is handed a read-only token on purpose:
# the code being tested is not yours. So the run that executes the tests
# writes the report and the PR number as artifacts, and a second workflow -
# triggered by `workflow_run`, running from your default branch, with a
# writable token and none of the fork's code - posts the comment.
name: tests
on:
pull_request:
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: prashanth-sams/pytest-html-reporter-action@v1
id: report
with:
tests: tests/
comment: 'false' # nothing to comment with, here
artifact-name: pytest-html-report
# Written after the tests, so the fork's code has had its turn - and
# checked again by the second workflow against the commit this run
# actually tested, because an artifact out of a job that ran somebody
# else's tests is a claim, not a fact.
- name: Keep the summary and the PR number for the next workflow
if: always()
run: |
set -euo pipefail
rm -rf comment
mkdir -p comment
printf '%s' "$PR_NUMBER" > comment/pr-number
printf '%s' "$SUMMARY" > comment/body.md
env:
PR_NUMBER: ${{ github.event.number }}
SUMMARY: ${{ steps.report.outputs.summary }}
- uses: actions/upload-artifact@v4
if: always()
with:
name: pytest-html-report-comment
path: comment/
Note the ordering: the two text files are written after the tests, so the fork’s code has already had its turn, and if: always() on both trailing steps means a red run still gets its comment.
# The second half of fork-pr.yml. Save this as its own workflow file.
#
# `workflow_run` runs from your default branch with your permissions, and
# never checks out the fork's code. It reads two small text files, and it
# checks that the pull request they name is really the one that produced
# them - an artifact comes out of a job that ran the fork's tests, so it is
# not on its own evidence of anything.
name: tests comment
on:
workflow_run:
workflows: [tests]
types: [completed]
permissions:
contents: read
actions: read
pull-requests: write
jobs:
comment:
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v4
with:
name: pytest-html-report-comment
path: comment
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ github.token }}
- uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const marker = '<!-- pytest-html-reporter-action -->';
const run = context.payload.workflow_run;
const claimed = Number(fs.readFileSync('comment/pr-number', 'utf8').trim());
if (!Number.isInteger(claimed) || claimed <= 0) {
core.setFailed('the artifact did not carry a usable pull request number');
return;
}
// The artifact was produced by a job running the fork's code, so
// the number in it is a claim rather than a fact. A pull request
// whose head is the commit this run tested is the one thing that
// cannot be forged from inside that job.
const { data: pull } = await github.rest.pulls.get({
...context.repo, pull_number: claimed,
});
if (pull.head.sha !== run.head_sha) {
core.setFailed(
`pull request #${claimed} is at ${pull.head.sha}, but this run ` +
`tested ${run.head_sha}. Refusing to comment on a pull request ` +
`the artifact only claims to belong to.`);
return;
}
const body = marker + '\n'
+ fs.readFileSync('comment/body.md', 'utf8').slice(0, 60000);
const existing = await github.paginate(github.rest.issues.listComments, {
...context.repo, issue_number: claimed, per_page: 100,
});
const mine = existing.find((comment) => comment.body?.includes(marker));
if (mine) {
await github.rest.issues.updateComment({
...context.repo, comment_id: mine.id, body,
});
} else {
await github.rest.issues.createComment({
...context.repo, issue_number: claimed, body,
});
}
pull.head.sha equals workflow_run.head_sha — the commit this run actually tested. That equality is the one thing that cannot be forged from inside the untrusted job.Everything the tests print is untrusted text
On a fork’s pull request, test names and assertion messages belong to the fork author, and they end up in a job summary and in a comment posted under the repository’s own identity. The action hardens every path they travel.
- The pytest output is wrapped in
::stop-commands::with a token of 16 random bytes from/dev/urandom, so nothing pytest prints is read as a workflow command — not::add-mask::, and not a::stop-commands::of its own. - Names are escaped three ways: HTML, because a failure sits inside a
<details>block and a name of</details>would close it and take the rest of the summary with it; markdown brackets, so a name cannot become a link; and, inside a table, the pipe, because GFM ends a cell at an unescaped one however deeply nested. - Lines that open with
::are defanged with a zero-width space. - Captured messages are fence-safe and trimmed to 1,200 characters.
$GITHUB_OUTPUTheredoc delimiters are random, so a test name carrying the wordEOFcannot close the block early.
Build history
Reach for this when you have a scheduled or repeated run and want to see change over time. The report’s Trends, Archives and Analytics tabs are built from previous builds, which a fresh runner does not have. history: 'true' carries them between runs with the Actions cache.
# Trends, Archives and Analytics need more than one build to say anything.
# The cache carries the archived builds from run to run; the retention
# limits stop the report growing without end.
name: nightly
on:
schedule:
- cron: '0 2 * * *'
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: prashanth-sams/pytest-html-reporter-action@v1
with:
tests: tests/
title: Nightly
history: 'true'
archive-days: '30'
environment: staging
What is cached
Both <report-dir>/archive and <report-dir>/output.json. Carrying only archive/ accumulates nothing at all — a build joins the archive only when the next run rotates its output.json in, so the cache has to carry that json too.
The cache key is <history-key>-<runner.os>-<github.ref_name>-<github.run_id>-<github.run_attempt>, with restore-keys falling back first to the branch prefix and then to the OS prefix. So history follows a branch where it can and falls back to the same OS otherwise, and every run writes a new immutable cache entry rather than trying to overwrite one.
What the prime step fixes before pytest runs
A cache restore on its own is not enough, because two things have to be true for a build to join the archive.
- A placeholder report file. The plugin archives a build by rotating the previous
output.jsonintoarchive/, and it only does that when the previous run’s report file is on disk. On a fresh runner it never is, so every run would quietly replace its predecessor and the archive would stay empty for ever. The action stands in an empty file and says so; the run then overwrites it. - Quarantining unreadable builds. The plugin reads every
archive/*.jsonwithout guarding the read, so a single truncated or foreign file raises insidepytest_terminal_summaryand no report is written at all — and a restored cache is exactly where such a file comes from. Each one is validated, and a file that fails is renamed to<name>.json.unreadable, with a warning giving the count and the reason.
Because the restore puts the previous build’s output.json on disk before pytest runs, a run that then writes none of its own — it crashed, it collected nothing — would leave the previous build there to be summarised. The action compares the file’s start_time against the restored one and, if they match, reads it as no report at all rather than reporting somebody else’s passes as this run’s.
Keeping the report from growing without end
Every retained build costs roughly 5KB of the page, so an hourly run with no limit reaches a multi-megabyte report within a couple of months and the page gets slow to open. Three inputs prune it, and they intersect — a build has to satisfy every limit you set to be kept: archive-count, archive-days and archive-since.
report-path containing strftime placeholders makes a new folder every run, so every run starts from nothing — put the date in artifact-name instead. And a matrix needs a history-key per cell, or two Python versions interleave builds that are not comparable.The first run with history on still shows one build. It takes two.
GitHub Pages
Reach for this when you want a URL anyone can open, with no artifact to download and unzip. pages-artifact: 'true' runs actions/upload-pages-artifact@v3 against the report directory. That is as far as an action can take you: deploying to Pages needs its own job with the github-pages environment and pages: write plus id-token: write, and an action cannot grant itself either.
# Publishing the report to GitHub Pages.
#
# The deploy has to be its own job: it needs the github-pages environment
# and its own permissions, neither of which an action can grant itself.
name: tests
on:
push:
branches: [main]
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: prashanth-sams/pytest-html-reporter-action@v1
with:
tests: tests/
pages-artifact: 'true'
history: 'true'
# The report is one page, so name it index.html and Pages serves
# it at the root of the site.
report-path: report/index.html
deploy:
needs: test
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deploy.outputs.page_url }}
steps:
- uses: actions/deploy-pages@v4
id: deploy
Two things in that workflow are load-bearing. report-path: report/index.html, because the report is one page and naming it index.html makes Pages serve it at the root of the site; and history: 'true', because a published report is exactly where Trends and Archives earn their keep. Once the site is up, feed its URL back in with report-url and the job summary and pull request comment carry an Open the report link straight to it.
Thresholds and gates
Reach for a hard floor when the rule is simple enough to state as a number. Both gates are enforced by the action itself.
- uses: prashanth-sams/pytest-html-reporter-action@v1
with:
tests: tests/
min-pass-rate: '95'
min-coverage: '80'
The rules, in order
- Exit code 5 — no tests collected — is handled first and on its own, by
fail-on-empty.fail-on-erroris not consulted for code 5. - Any other non-zero exit code fails when
fail-on-erroris true, with the code spelled out in the annotation. - A clean exit with no report also fails when
fail-on-erroris true. min-pass-rate, when set and numeric. If the pass rate cannot be measured at all the run fails rather than passing by default.min-coverage, when set and numeric. A run that produced no coverage data fails.
Every reason that applies is collected. Each is emitted as an ::error title=pytest-html-reporter:: annotation, and a > [!CAUTION] blockquote listing all of them is appended to the job summary.
min-coverage on a run that produced no coverage is an error, not a free pass. The same rule governs min-pass-rate on a run where nothing decisive happened.Pass rate is deliberately narrow
pass-rate = passed / (passed + failed + errors), as a percentage. Skipped, xfailed and xpassed tests are in neither half — none of them is a pass-or-fail signal, and counting them would quietly move the threshold you set. When the denominator is zero, pass-rate is empty and any min-pass-rate fails.
Exit code against report status
The plugin calls a run FAIL only when a suite holds a failure or an error. pytest has more ways to exit non-zero than that: a strict xfail, pytest.exit(), an interrupted run. The exit code is what the job is decided on, and status is the report’s headline. When the two disagree the action says so rather than picking one quietly.
| Exit code | What it means |
|---|---|
1 | Tests failed. |
2 | The run was interrupted. |
3 | An internal error. |
4 | A usage error: pytest refused an argument. Nothing ran. |
5 | No tests were collected. Governed by fail-on-empty, not fail-on-error. |
Deciding for yourself
Reach for this when “any red test fails the build” is too blunt — a few flakes are survivable, a collapse is not. fail-on-error: 'false' lets the red run through, and a following step reads the outputs and makes the call.
# Deciding for yourself what a failing run means.
name: tests
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: prashanth-sams/pytest-html-reporter-action@v1
id: report
with:
tests: tests/
# Let a red run through, and judge it below.
fail-on-error: 'false'
# != 'PASS' rather than == 'FAIL': a run that produced no report at all
# reports UNKNOWN, and that is the one outcome nobody should wave through.
- name: A few flakes are survivable; a collapse is not
if: steps.report.outputs.status != 'PASS'
run: |
set -euo pipefail
if [ "$STATUS" != "FAIL" ]; then
echo "::error::the run produced no report to judge (status $STATUS)"
exit 1
fi
echo "pass rate: $PASS_RATE% ($FAILED failed of $TOTAL)"
if [ -z "$PASS_RATE" ] || [ "${PASS_RATE%%.*}" -lt 90 ]; then
echo "::error::pass rate $PASS_RATE% - this is not flakiness"
exit 1
fi
echo "::warning::$FAILED failing, pass rate still $PASS_RATE% - letting it through"
env:
STATUS: ${{ steps.report.outputs.status }}
PASS_RATE: ${{ steps.report.outputs.pass-rate }}
FAILED: ${{ steps.report.outputs.failed }}
TOTAL: ${{ steps.report.outputs.total }}
Two details there are deliberate and worth copying. The condition is != 'PASS' rather than == 'FAIL', because a run that produced no report at all reports UNKNOWN and that is the one outcome nobody should wave through. And every value reaches the script through env: rather than being spliced into the command.
Coverage
Reach for this when you already measure coverage with pytest-cov and want it in the report and enforced. Nothing here is special to the action: install pytest-cov, pass --cov, and the plugin reads the coverage the run just measured.
# The Test Coverage tab, and a floor under it.
#
# Nothing here is special: install pytest-cov, pass --cov, and the plugin
# reads the coverage the run just measured.
name: tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: prashanth-sams/pytest-html-reporter-action@v1
with:
python-version: '3.12'
extra-packages: pytest-cov
tests: tests/
pytest-args: --cov=src --cov-branch
min-coverage: '80'
The plugin reads whatever measured coverage; it does not measure any itself. Auto-discovery looks for a coverage.json or coverage.xml in the report directory, the repository root and the working directory. A .coverage data file is not discovered — name it with report-coverage-file, and install coverage in that job so it can be read.
coverage.xml restored from a cache before the tests ran is published as this build’s number, silently. Do not cache coverage files alongside the report.Matrix and sharding
Reach for this when you test across operating systems and Python versions. The action is designed to be run once per matrix cell, and produces one report per cell.
# One report per cell. Artifact names must differ within a run.
name: tests
on: [push, pull_request]
jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python: ['3.9', '3.11', '3.13']
steps:
- uses: actions/checkout@v4
- uses: prashanth-sams/pytest-html-reporter-action@v1
with:
python-version: ${{ matrix.python }}
tests: tests/
title: ${{ matrix.os }} py${{ matrix.python }}
artifact-name: pytest-html-report-${{ matrix.os }}-py${{ matrix.python }}
build-info: |
os=${{ matrix.os }}
python=${{ matrix.python }}
- Artifact names must be unique within a run.
actions/upload-artifact@v4refuses two artifacts of the same name, so give each cell its ownartifact-name.artifact-overwrite: 'true'exists, but overwriting means eight of your nine reports are gone. - Label the reports.
titlesets the heading inside the report andbuild-infoaddsKEY=VALUErows to it, so a downloaded report says which cell produced it. - History needs a key per cell. The default cache key already includes
runner.os, but two Python versions writing into one history interleave builds that are not comparable. fail-fast: falseis what you want here: a report for the cell that failed is more useful than a cancelled matrix.
Sharding within a job is a pytest-args concern rather than an action concern. Splitting a suite across jobs produces one report per job, each with its own artifact; the action does not merge them.
- uses: prashanth-sams/pytest-html-reporter-action@v1
with:
tests: tests/
extra-packages: pytest-xdist
pytest-args: -n auto -m "not slow"
Permissions
The action needs nothing beyond the default for the basic case. Add what the features you turn on need. Since 2023 the default GITHUB_TOKEN is read-only, so a missing pull-requests: write is the usual reason a comment does not appear.
permissions:
contents: read # always
pull-requests: write # for `comment: 'true'`
| Permission | Where it goes | Why |
|---|---|---|
contents: read | The test job | Always. Checkout. |
pull-requests: write | The test job | Required for comment: 'true'. Unavailable to a pull_request run from a fork. |
actions: read | The workflow_run comment job | To download the artifact produced by the test run. |
pages: write | The deploy job only | For actions/deploy-pages. |
id-token: write | The deploy job only | For actions/deploy-pages. |
How it works
The action is a composite: thirteen steps in action.yml, and one stdlib-only Python helper, scripts/phr.py, that does the thinking.
| Step | What it does |
|---|---|
| Check the inputs | Validate before anything expensive happens. |
| Set up Python | actions/setup-python, only when python-version is set. |
| Find the interpreter | python or python3 — settle on one name. |
| Install | The plugin, your requirements, extra packages. |
| Work out where | Resolve report-path exactly as the plugin will. |
| Restore build history | actions/cache/restore, when history is true. |
| Run pytest | Prime, probe, run, capture the exit code. |
| Save build history | actions/cache/save, when history is true. |
| Upload the report | actions/upload-artifact. |
| Upload a Pages artifact | actions/upload-pages-artifact, when pages-artifact is true. |
| Summarise the run | output.json to outputs, job summary and comment body. |
| Comment | actions/github-script, when comment is true. |
| Decide the job | Go red, or do not. |
The helper is invoked four times.
| Subcommand | What it does |
|---|---|
resolve | Expands strftime placeholders once, normalises backslashes, applies the plugin’s file-or-folder rule, creates the directory, and emits the path outputs. |
prime | Readies a restored cache: records the restored build’s start_time, stands in an empty report file so the predecessor joins the archive, quarantines unreadable archived builds, and deletes stale pytest_screenshots. |
args | Turns the PHR_* environment into a pytest argument list, filtered against pytest --help, written NUL-separated for bash to read back. |
summarize | Reads output.json and produces the step outputs, the job summary markdown, the comment body and the threshold verdict. |
gate-passed back out of the summarize step. true passes and false fails — and anything else, including empty, also fails, because a run nobody checked is not a run that passed.Why the path is resolved before pytest runs
--html-report is run through strftime, so reports/%H%M/ names a different folder each minute. If the action expanded it separately from the plugin, a run crossing a minute boundary would leave the action looking in a folder the report is not in. So the action expands it once, up front, and hands pytest an already-concrete path — with every remaining percent doubled, because the plugin expands whatever it is given and a surviving % would be read as a directive on the second pass.
The file-or-folder rule is copied from the plugin rather than reinvented, including its sharp edge: the value names a file when it contains .html anywhere, not when it ends with it. When no .html is present, the report is named pytest_html_report.html. On Windows the action normalises backslashes to forward slashes first, because the plugin splits the value on / only.
The resolve step emits report-dir, report-name, report-file, json-path, archive-dir, screenshot-dir and html-report, and warns when the resolved directory is the working directory — because then your whole checkout is what gets uploaded as the artifact.
Why pytest’s flags are probed first
Anyone can pin their own version of the plugin, and the older releases have fewer options. An unknown flag is not a missing feature — argparse aborts the entire run with exit code 4 before a single test is collected, naming a flag the user never typed.
So the action runs pytest --help once and passes only the flags that appear in it. The check requires --html-report to be present as evidence that the probe is real: a probe that did not actually list the plugin’s options would otherwise read as “this plugin supports nothing” and drop every flag. If the probe itself fails, nothing is filtered — passing the flags and letting pytest object beats dropping all of them. Anything left out is announced as a warning naming the flag.
The argument list is assembled in a fixed order: --html-report first and always; then the single-value flags, each included only when its input is non-empty; then the repeated flags (--build-info, --report-link), one per non-blank line; then --report-open; then each line of tests; then pytest-args. It is written NUL-separated to a temp file and read back into a bash array, so no argument can be re-split or globbed on the way, and the assembled command is printed shell-quoted so the log shows exactly what ran.
Why nothing is interpolated into the shell
Every input reaches bash as an environment variable, never as ${{ ... }} spliced into a run: body — that would make pytest-args shell source. tests/test_action_yml.py fails the build if anyone writes one. Test names and failure messages get the same treatment on the way out; see untrusted text.
Troubleshooting
Most of what goes wrong here is one of a handful of things. If the problem is in the report’s contents rather than in getting it built, it probably belongs on the plugin’s side.
No output.json was written
The run finished and the plugin never wrote anything. Three causes, in order of likelihood.
The plugin is not installed in the environment that ran the tests. The action’s install step and your test run must use the same Python. If you set install: 'false' because your requirements.txt pins it, check that the file really lists pytest-html-reporter. The action logs a warning at install time when the import fails — look for it above the pytest output.
pytest failed before any test ran. A collection error, a missing dependency, a bad -k. The pytest log says which.
report-path points somewhere else. Check the report-dir output against where you are looking.
Exit code 4: “a usage error”
pytest refused an argument. This is not a failing test — nothing ran.
The message pytest printed names a flag, and you set an input. Nearly all of them share a name — --archive-since is the archive-since input — with two that do not: --html-report is report-path, and --report-link is report-links. The same message also appears when the value came from your pytest.ini, because the ini keys share their names with the flags too. So check both.
The action quotes the offending line back as an annotation, so you should not have to go digging in the log for it.
“the installed pytest-html-reporter has no --report-steps”
You are on a version of the plugin older than the input you set. The action leaves the flag out rather than failing the run — passing an unknown flag would abort pytest with a usage error before a single test was collected. Fix it by upgrading, or drop the input.
with:
plugin-version: '>=0.3.8'
The Trends, Archives and Analytics tabs are empty
They read previous builds, and a fresh runner has none. Turn history on:
with:
history: 'true'
Then check three things.
report-pathmust be fixed, not per-run. A path with%Y%m%din it makes a new folder every run, so every run starts from nothing. Put the date inartifact-nameinstead. The action warns when it sees this combination.- A matrix needs a
history-keyper cell. The default key already includes the runner OS, but two Python versions writing into one history interleave builds that are not comparable. archive-count: '1'keeps nothing. The plugin reads it as “this build and no others”.'0'deletes the archive entirely. Leave it empty for no limit.
Note that the first run with history on still shows one build. It takes two.
The Test Coverage tab is empty
The plugin reads whatever measured coverage; it does not measure any itself.
- Install
pytest-covand pass--cov=<your package>inpytest-args, in the same run. That is the whole of it for a single job. - Auto-discovery looks for
coverage.jsonorcoverage.xmlin the report directory, the repository root and the working directory. A.coveragedata file is not discovered — name it withreport-coverage-file, and installcoveragein that job so it can be read. report-coverage-fileis final. If the file named there cannot be read, the tab stays blank and the coverage the run measured is not used instead. The action warns when this happens.- There is no freshness check. A
coverage.xmlrestored from a cache before the tests ran is published as this build’s number, silently. Do not cache coverage files alongside the report.
The pull request comment never appears
- Permissions.
pull-requests: writeon the job. The default token has been read-only since 2023. - A fork. A
pull_requestrun from a fork gets a read-only token and cannot comment at all. This is deliberate on GitHub’s part. Use theworkflow_runpattern in fork pull requests. - The event.
comment: 'true'only fires on apull_requestevent, unless you passpr-numberexplicitly.
The action warns and carries on rather than failing the run, so look for a warning annotation rather than a red step.
The report is enormous
Every archived build costs roughly 5KB of the page. An hourly run with no retention limit reaches several megabytes within a couple of months, and the page gets slow to open.
Set archive-days: '30', or archive-count: '50'. They intersect with archive-since — a build has to satisfy every limit you set to be kept.
report-log-limit, report-attachment-limit and report-step-limit cap what each test contributes. Note that 0 means unlimited for all three, and a negative value is read as 0 — the action rejects a negative rather than letting it silently mean the opposite of what it looks like.
The job went green and no tests ran
That is what fail-on-empty is for, and it is on by default — a run that collects nothing fails. If a cell of your matrix legitimately has no tests, set fail-on-empty: 'false' for it.
The report landed in the wrong folder
report-path is read as a file when it contains .html anywhere, not just at the end. So out.html_v2/report.html makes the plugin treat the whole thing as a bare filename and write into the working directory. Rename the folder. The action warns when it detects this.
Percent signs are strftime placeholders: %p is AM/PM, so ./100%pass/ becomes ./100PMass/. Write %% for a literal percent.
pip refuses to install anything
error: externally-managed-environment. The runner’s Python belongs to its distribution, and pip will not touch it. This does not happen on GitHub-hosted runners; on a self-hosted one, set python-version so actions/setup-python provides an interpreter of its own.
with:
python-version: '3.12'
Something else
Open an issue with the workflow YAML, the job log, and the report-dir and status outputs. If the problem is in the report’s contents rather than in getting it built, it probably belongs on the plugin — start with the FAQ and the CLI reference.