Running tests in CI — GitHub Actions, reports, and failure artifacts

Why this matters

A test suite that only runs on developers' laptops is not automated — it's manual with extra steps. The value of test automation is realized when tests run automatically on every code change, without anyone having to remember to run them. GitHub Actions makes this a 20-line YAML file: on every pull request, checkout the code, install dependencies, run the tests, and upload the Playwright HTML report as a build artifact. When tests fail in CI, the failure artifact (screenshot, video, trace) is the evidence you need to diagnose and fix the issue — without it, 'tests failed in CI' is nearly as useless as 'it doesn't work'.

Demo

GitHub Actions turns a Playwright suite into a fully automated quality gate by running tests on every pull request and uploading the HTML report and failure traces as artifacts regardless of outcome. Without artifact upload, a CI failure leaves only a log line to diagnose — with it, the exact screenshot, video, and network timeline of the failure are one click away. Configuring this correctly is the difference between CI that surfaces bugs and CI that silently blocks merges.

// .github/workflows/playwright.yml
// Push this to your repo and it runs on every PR

/*
name: Playwright Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium

      - name: Run Playwright tests
        run: npx playwright test
        env:
          BASE_URL: ${{ secrets.STAGING_URL }}   # inject env vars from repo secrets

      # Always upload the report — even on failure (that's when you need it most)
      - name: Upload Playwright report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14   # keep for 14 days then auto-delete

      # On failure: upload full trace for debugging
      - name: Upload trace on failure
        uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-traces
          path: test-results/
*/

// playwright.config.ts — configure traces and videos
export default {
  use: {
    baseURL: process.env.BASE_URL || 'http://localhost:3000',
    trace: 'on-first-retry',        // capture trace on first retry
    video: 'retain-on-failure',     // save video only when test fails
    screenshot: 'only-on-failure',  // save screenshot only on failure
  },
  reporter: [['html', { open: 'never' }], ['github']],  // GitHub annotations
};

Try it yourself

1. Create a GitHub repository, add a minimal Playwright test, and push the playwright.yml workflow. Go to the Actions tab and watch the workflow run. Download the uploaded HTML report. This is the CI loop that runs on every PR.
2. Intentionally break one of your tests (change an expected value). Push the change. In the Actions tab, find the failed workflow run and download the failure artifact. Open the trace file in npx playwright show-trace trace.zip. This is how you debug CI failures without running locally.
3. Add a matrix strategy to the workflow: run tests on multiple browsers: matrix: browser: [chromium, firefox, webkit]. How does the total run time change? How would you decide which browsers to include in CI?
4. Set up a Slack/Discord notification: use slackapi/slack-github-action to post a message when the workflow fails. This models a real team's alerting setup — test failures should be visible, not hidden in a CI dashboard no one checks.

Prompt your AI

Use these three in order. Each builds on the one before.

1. Basics & terminology

In one paragraph, explain why test automation only delivers its full value when tests run in CI rather than on developer laptops. What are the two categories of things CI catches that local runs don't?

2. Why it works (the mechanism)

Walk me through what `if: always()` does in a GitHub Actions step and why it's used for the artifact upload step. What happens to the report if you use `if: success()` instead?

3. Advanced — application & what's next

My Playwright CI job takes 18 minutes. Our PRs wait in a queue for 25 minutes before getting test results, slowing down the team. Walk me through four specific optimizations: test sharding, browser subset selection for PR vs merge, caching strategies, and parallelization. Estimate time savings for each.

References

• Playwright — GitHub Actions
• GitHub Actions documentation