Compatibility & files
What is actually tested rather than what setup.py declares, which plugins the reporter understands, exactly what a run leaves in your repository, and the full licence inventory of the ten libraries and one icon set that ship inside every generated report.
Versions and platforms
Nothing in the three repositories states a support policy, and the one machine-readable claim
is now closer to the truth than it was. setup.py declares
python_requires=">=3.7" as of 0.4.1, raised from >=3.5
because the async step support added in that release needs contextvars, which is 3.7.
The package also uses f-strings — in cli.py, junit.py,
coverage_report.py and report_opener.py — and nothing in CI has exercised
3.7 or 3.8 in years. What follows is the tested matrix rather than the declared one.
| What | Tested | Ought to work | Where that is tested |
|---|---|---|---|
| Python | 3.9 and 3.13 | 3.8 and newer | The Action's .github/workflows/ci.yml matrix |
| Operating system | Ubuntu, macOS, Windows | Anything Python runs on | The Action's ci.yml and self-test.yml matrices |
| pytest | Whatever pip install pytest resolves to |
Current majors, with the current plugin release | The plugin's .github/workflows/main.yml |
Python
The Action's own CI matrix runs 3.9 and 3.13 on ubuntu-latest,
macos-latest and windows-latest — six cells, fail-fast:
false, so a break on one interpreter does not hide a break on the other. The plugin's own
workflow (main.yml) runs the unit suite on whatever Python ubuntu-latest
provides, on pull requests to master only.
Treat 3.8 as the floor and 3.9–3.13 as the tested band —
a step above the >=3.7 that setup.py declares, because 3.7 is what the
code needs rather than what anything runs it on. If you need an older interpreter, pin an older
plugin release and test it against your own suite — nobody else is.
tox.ini still lists
py37 and pypy3, and an ansi2html factor that no code in the
package reads. It is stale; the GitHub workflows are the live description of what gets run.pytest
install_requires is an unpinned pytest, and the plugin tracks pytest
closely rather than pinning it. Release 0.3.0 exists partly because pytest 9 removed
TerminalReporter._sessionstarttime, which older releases reached into — the changelog
entry reads "Fixed compatibility with pytest 9.x (AttributeError: '_sessionstarttime' not
found)", and the reporter now keeps its own _sessionstarttime instead.
That is the shape of the risk. A pytest major release can break an old plugin release, and the fix ships as a new plugin release rather than as a version pin. Upgrade the plugin when you upgrade pytest.
Operating systems
setup.py classifies the package Operating System :: OS Independent, and
the Action's self-test.yml backs that up by running the whole composite on
ubuntu-latest, macos-latest and windows-latest and asserting
on every output.
One behaviour is genuinely platform-sensitive: opening the report. macOS and
Windows are assumed to have a desktop. Everywhere else, DISPLAY or
WAYLAND_DISPLAY has to be set, or --report-open=auto stays quiet — see
CI integrations for the full check.
The three projects
| Project | Version | Requires |
|---|---|---|
| Plugin | 0.4.3 |
Python 3.8+, pytest, Pillow |
| GitHub Action | v1, moved on each release |
Any runner with Python; the helper is stdlib-only |
| VS Code extension | 0.1.2 |
VS Code ^1.90.0 |
The three version numbers are deliberately independent; there is no relationship between them.
The extension reads output.json and nothing else, so it works with any plugin release
that writes one — but its flaky-test verdicts need archive/ populated, which needs at
least two builds.
Dependencies
Two hard dependencies, and that is the whole of it. Everything else is detected at run time and degrades to "that tab is empty" rather than failing the run.
Required
| Package | Needed for |
|---|---|
pytest |
The plugin is a pytest plugin. Unpinned, so pip resolves whatever your project already has. |
Pillow |
Writing and handling the screenshot images the reporter captures. |
Optional — each unlocks one thing
None of these are installed for you, and none of them are imported unless they are present.
| Package | Needed for | What happens without it |
|---|---|---|
pytest-cov |
The Test Coverage tab, read from the Coverage object pytest-cov ran in the
same session. |
Tab is empty, unless you point --report-coverage-file at a report of your
own. |
coverage |
Reading a bare .coverage SQLite data file. Only auto-discovered names are
coverage.json and coverage.xml; a .coverage has to be
named explicitly and needs this package. |
summarize_data_file returns None and the tab stays blank. |
pytest-xdist |
Running -n auto. Workers are detected by config.workerinput and
ship their records back to the controller, which writes one report. |
Nothing — the run is simply not parallel. |
pytest-bdd |
Gherkin scenarios arriving in the Test Steps tab, with feature, scenario, keyword and parsed arguments. | Nothing. The hooks are declared optionalhook, so pytest does not refuse to
start over a hook nobody registered. |
pytest-rerunfailures |
Folding a retried test's attempts into one row, and counting a retry as flake evidence.
Detected with pluginmanager.hasplugin("rerunfailures"). |
Nothing — there are no reruns to fold. |
| A browser driver | Automatic screenshots on failure. Selenium, Playwright, splinter, seleniumbase and pytest-selenium are all recognised. | The Screenshots tab is empty. |
requirements.txt is the development set — pytest-cov,
coveralls, twine, pytest-xdist,
beautifulsoup4, pytest-rerunfailures, Pillow and
pytest-bdd. Do not read it as the runtime set; that is
install_requires.Plugins and drivers it understands
Recognised explicitly
Four plugins are handled by name in the source, and each one is handled defensively — the reporter asks pytest's plugin manager whether they are there rather than importing them.
-
pytest-xdistWorkers are told apart by
hasattr(config, "workerinput"). Each worker fills its own record list and ships it to the controller, which writes one report — one build, in collection order. Nothing to configure. - pytest-rerunfailuresAttempts are folded into one row only when the plugin is actually installed, because pytest-rerunfailures runs the whole setup/call/teardown protocol again and the fold would otherwise be guessing.
-
pytest-bdd 4 through 8Every object handed to the BDD hooks is read with
getattracross several candidate names, because the attributes moved between major versions and this plugin pins neither. -
pytest-covIts
Coverageobject is read out of the plugin manager (getplugin("_cov")) after it has stopped, saved and combined — read rather than raced for.
Browser drivers, recognised by shape
No driver is imported and no driver is named as a dependency. The reporter looks through the
test's own fixtures for a value that can be photographed, trying get_screenshot_as_png
first and then screenshot, and looking one level down at a .driver
attribute for wrappers. The fixture names it searches are the ones real suites use:
| Fixture name | Usually from |
|---|---|
page, context | pytest-playwright |
driver, webdriver | Selenium, by convention |
browser | splinter, Playwright |
session_browser | pytest-splinter |
selenium | pytest-selenium |
sb | seleniumbase |
The same duck typing runs on HTTP clients: attach_api reads a response by shape, so
requests and httpx both work without either being imported.
Mock is ruled out
by what it is, not by what it can do — it answers every call ever made to it, screenshot
calls included, so it would be photographed on the strength of a method it does not have, and the
call would show up in an assertion your test is about to make.Not supported
| Thing | Why not |
|---|---|
behave |
It does not run under pytest, so there is no hook to implement and nothing that could be shown. Only pytest-bdd is covered. |
-s / --capture=no |
Not a plugin, but the most common cause of an empty Logs column. pytest has to be
capturing for there to be anything to keep. Use --capture=tee-sys to get output
on the terminal and in the report. |
| Two report writers on one folder | A report folder has exactly one writer per build. That is also why the merge of a sharded run is a console script rather than another pytest run — a fifth pytest started in the folder would clear away the screenshots it was sent to collect. |
What a run writes to disk
This is the first thing a newcomer hits, and the README currently gets it wrong. The Usage
section says "the filename used is pytest_html_reporter.html and path chosen
is report". Neither half is true. With no --html-report, the
plugin writes pytest_html_report.html — one er shorter — into the
current working directory, which is usually your repository root:
return os.path.abspath(logfile), 'pytest_html_report.html'
Run pytest once in a fresh directory and you get three things beside your tests: the
report, an output.json, and — from the second run onwards — an archive/
folder. Nothing is cleaned up and nothing warns you. So the first thing to do in a real project is
give the report a folder of its own and tell git to ignore it.
pytest in a clean project, then ls -la showing pytest_html_report.html, output.json and archive/ sitting in the repository root beside the tests. The file size on the HTML should be legible — it is the 1.6 MB figure made concrete.
Everything a run can create
All paths are relative to the report base — the folder side of
--html-report. With no flag, the report base is the working directory.
report/
├── pytest_html_report.html the report itself
├── output.json this build's record
├── archive/
│ ├── output_1756900000.123.json a previous build, rotated in
│ └── output_1756903600.456.json
├── pytest_screenshots/
│ └── <test name>.png images the report links to
└── shards/ only in a sharded run
└── <shard id>/
├── records.json this leg's records, written atomically
└── pytest_screenshots/ this leg's images, left where they are
| Path | Written when | What it is |
|---|---|---|
pytest_html_report.html |
Every non-shard run | The whole report: markup, styles, data and every vendored library, in one file. Roughly 1.6 MB for a one-test run — that is the floor, not the content. |
output.json |
Every non-shard run | This build's record. The stable, documented build format the VS Code extension and the Action's helper both read. |
archive/ |
From the second run onwards | Previous builds. A build joins the archive when the next run rotates it there,
renaming output.json to output_<start time>.json. Each
retained build costs roughly 5 KB of the report page. |
pytest_screenshots/ |
When a screenshot is captured | PNGs. The report stores a bare name and both screenshot templates rebuild the path, so the folder has to travel with the HTML. |
shards/<id>/ |
--report-shard only |
One directory per leg, holding records.json and that leg's own images. A
shard writes no report and no output.json, and rotates no archive. |
| Your JUnit XML path | --report-junit only |
Written wherever you point it; parent directories are created. An unwritable path is a warning on stderr, never a failed build. |
This project's own report, with 296 archived builds, is 6.6 MB. If yours is heading that way, the
retention flags in the CLI reference are what you want —
--archive-days, --archive-count and --archive-since.
htmlcov/ is
not written by this plugin. When it exists and was written after this run started, the
report links to it relatively rather than copying it in — so the two travel together or the link is
simply not offered.Two path rules that surprise people
.html anywhere makes it a filename. The check is if '.html' in
self.path, not endswith. So
--html-report=./out.html_v2/report.html is read as a bare filename and the report lands
in the working directory, because the code then asks whether the directory part also
contains .html and falls back to . when it does. Rename the folder.
~ and $VAR are expanded, by
os.path.expanduser and os.path.expandvars — so
--html-report='$CI_PROJECT_DIR/report' and --html-report=~/reports both
work, on top of the strftime expansion documented in
Configuration.
The .gitignore block to paste
Both companion repositories ship a version of this. The Action's is the one to copy, because of
the leading slashes: they anchor each entry to the repository root, so a tracked fixture called
output.json deeper in your tree survives.
# Everything a pytest-html-reporter run leaves behind lives in one folder.
/report/
# What a pytest-html-reporter run leaves behind. Anchored to the root so a
# tracked fixture called output.json deeper in the tree is still tracked.
/pytest_html_report.html
/output.json
/archive/
/pytest_screenshots/
/report/
/reports/
What ships inside the report
Every generated report is a single self-contained file, and that is only possible because ten JavaScript and CSS libraries plus a Font Awesome icon set are vendored into the package and inlined at generation time. If you are reviewing this plugin for a licence audit, this section is the inventory.
Why they are vendored at all
The report used to pull these off a CDN. A run on a machine with no route to the internet produced
a blank white page: jQuery never arrived and nothing after it ran. They now ship in
the wheel, verbatim as published, and html_page/assets.py writes them into the report.
That is what makes the offline guarantee real rather than aspirational — you can mail the file,
attach it to a ticket, put it on a stick, or open it from a locked-down build agent, and every
chart, table, filter and export still works.
The inventory
Every one of them is MIT. JSZip is dual MIT/GPLv3 and is used under MIT.
| File | Version | Licence |
|---|---|---|
jquery.min.js | jQuery 3.5.1 | MIT |
jquery.dataTables.min.js / .css | DataTables 1.10.19 | MIT |
dataTables.buttons.min.js, buttons.dataTables.min.css | Buttons 1.5.2 | MIT |
buttons.html5.min.js, buttons.print.min.js | Buttons 1.5.2 | MIT |
buttons.colVis.min.js | Buttons 1.6.1 | MIT |
bootstrap.min.js / .css | Bootstrap 4.1.3 | MIT |
chart.min.js | Chart.js 2.8.0 | MIT |
jspdf.min.js | jsPDF 1.3.2 | MIT |
dom-to-image.min.js | dom-to-image 2.6.0 | MIT |
jszip.min.js | JSZip 3.1.3 | MIT or GPLv3 (used under MIT) |
Icons
Font Awesome 4.7.0 glyphs, kept as individual SVGs, sourced from
Font-Awesome-SVG-PNG
(black/svg). Licence: CC BY 4.0, per the Font Awesome 4.7.0
licence.
Each file keeps its Font Awesome name (fa-home → home.svg) and is
cropped to the glyph's own outline, with the viewBox still recording that outline's
place on the font's 1792-unit em — the em Font Awesome drew on, with the baseline 1536 units down.
html_page/icon_styles.py reads the directory at generation time and inlines every glyph
as a CSS mask, which is why <i class="fa fa-home"></i> still
works, an icon still sizes itself to the text around it, and it still takes its colour from
currentcolor.
The one library deliberately absent
template.html.If you are vendoring a replacement for anything, this is the precedent: check the licence before the feature.
Upgrading one, or adding an icon
Replace the file, update the row in html_page/vendor/README.md, and leave the load
order alone — it lives in assets.py, not in the README. To add an icon, drop its SVG
into html_page/icons/ named after the fa-* class and the stylesheet picks
it up automatically; crop it to the outline first, or it is drawn in the wrong place and at the wrong
size.
MANIFEST.in
controls the sdist and setup.py's package_data plus
include_package_data control the wheel. A new asset type has to be added in
both, or the sdist has it and the wheel does not — and a wheel missing its assets
generates a blank white page.Glossary
A dozen terms these docs use with precise meanings that are easy to guess wrong.
| Term | What it means here |
|---|---|
| Run | One pytest invocation. |
| Build | One row of history — one output.json. Retention, archives, trends and
analytics all count builds, never runs. |
| Report base | The folder side of --html-report. Every path above is relative to it. With
no flag it is the current working directory. |
| Archive | <base>/archive/. A build joins it when the next run renames
output.json to output_<start time>.json — so a build is
archived by its successor, not by itself. |
| Worker | An xdist process on this machine, identified by
config.workerinput. Workers hand their records to a controller, which writes one
report. Nothing to configure. |
| Shard | A separate machine or leg, writing <base>/shards/<id>/records.json
and no report of its own. Something has to merge the bundles afterwards. |
| Run token | What tells a merge which shards belong together. Given by
--report-shard-run, or derived from the CI system's own variables — always
carrying the name of the system it came from, so Jenkins build 41 and Drone build 41 cannot
collide. |
| Suite | The grouping the report's Suite Highlights use:
nodeid.split("::")[0], which is the test file. |
| Step | A named, timed piece of a test — @step("…") or with
step("…"), nesting. Setup, body and teardown are timed as steps whether or
not anyone declared any, and a pytest-bdd scenario arrives as steps for free. |
| Attachment | Text, JSON, a file or a whole HTTP request/response a test hands to the report. Built-in types only, because an xdist worker has to ship it back to the controller. |
| Decided build | A build in which the test did not skip. All the pass/fail maths — rate, flips, streak — is done over decided builds only, so a test skipped for three builds between two passes has not flipped twice. |
| Flaky | A test that flipped between pass and fail across its decided builds, or needed a retry inside a single build. A retry is the least ambiguous evidence there is: same code, same build, two different answers. |
| Standing failure | Shown as Always failing. Every decided build failed and the current failing streak is at least two builds — so one failing build is not yet one. A standing failure is explicitly not counted as flaky: it is broken, and listing it under flakiness sends people hunting for a race that is not there. |
Build vs. run
Usually they are the same thing — but an xdist run with eight workers is eight processes and one build, and a sharded matrix is four runs and one build. Everything about retention, archives and trends counts builds.
Shard vs. worker
Both split work up, and they compose, but they solve different problems and are handled completely
differently. Workers are processes on one machine that hand their results back to a
controller, which writes one report — nothing to configure. Shards are separate
machines, each writing a bundle and no report, and something has to merge them
afterwards. You can shard across four machines and run each shard with -n auto; see
CI integrations for the wiring.