Skip to main content

Continuous Integration

Playwright tests can be executed in CI environments. We have created sample configurations for common CI providers.

Introduction​

3 steps to get your tests running on CI:

  1. Ensure CI agent can run browsers: Use our Docker image in Linux agents or install your dependencies using the CLI.

  2. Install Playwright:

    pip install playwright
    playwright install --with-deps

  3. Run your tests:

    pytest

CI configurations​

The Command line tools can be used to install all operating system dependencies on GitHub Actions.

GitHub Actions​

steps:
  - uses: actions/checkout@v3
  - name: Set up Python
    uses: actions/setup-python@v4
    with:
      python-version: '3.10'
  - name: Install dependencies
    run: |
      python -m pip install --upgrade pip
      pip install -r local-requirements.txt
      pip install -e .
  - name: Ensure browsers are installed
    run: python -m playwright install --with-deps
  - name: Run your tests
    run: pytest

We run our tests on GitHub Actions, across a matrix of 3 platforms (Windows, Linux, macOS) and 3 browsers (Chromium, Firefox, WebKit).

GitHub Actions on deployment​

This will start the tests after a GitHub Deployment went into the success state. Services like Vercel use this pattern so you can run your end-to-end tests on their deployed environment.

name: Playwright Tests
on:
  deployment_status:
jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    if: github.event.deployment_status.state == 'success'
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v2
      with:
        node-version: '14.x'
    - name: Install dependencies
      run: npm ci
    - name: Install Playwright
      run: npx playwright install --with-deps
    - name: Run Playwright tests
      run: npx playwright test
      env:
        # This might depend on your test-runner/language binding
        PLAYWRIGHT_TEST_BASE_URL: ${{ github.event.deployment_status.target_url }}

Docker​

We have a pre-built Docker image which can either be used directly, or as a reference to update your existing Docker definitions.

Suggested configuration

  1. Using --ipc=host is also recommended when using Chromium—without it Chromium can run out of memory and crash. Learn more about this option in Docker docs.
  2. Seeing other weird errors when launching Chromium? Try running your container with docker run --cap-add=SYS_ADMIN when developing locally.
  3. Using --init Docker flag or dumb-init is recommended to avoid special treatment for processes with PID=1. This is a common reason for zombie processes.

GitHub Actions (via containers)​

GitHub Actions support running jobs in a container by using the jobs.<job_id>.container option.

steps:
  playwright:
    name: 'Playwright Tests'
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.27.1-focal
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r local-requirements.txt
          pip install -e .
      - name: Ensure browsers are installed
        run: python -m playwright install --with-deps
      - name: Run your tests
        run: pytest

Azure Pipelines​

For Windows or macOS agents, no additional configuration required, just install Playwright and run your tests.

For Linux agents, you can use our Docker container with Azure Pipelines support running containerized jobs. Alternatively, you can use Command line tools to install all necessary dependencies.

For running the Playwright tests use this pipeline task:

jobs:
    - deployment: Run_E2E_Tests
      pool:
        vmImage: ubuntu-20.04
      container: mcr.microsoft.com/playwright:v1.27.1-focal
      environment: testing
      strategy:
        runOnce:
          deploy:
            steps:
            - checkout: self
            - task: Bash@3
              displayName: 'Run Playwright tests'
              inputs:
                workingDirectory: 'my-e2e-tests'
                targetType: 'inline'
                failOnStderr: true
                env:
                  CI: true
                script: |
                  npm ci
                  npx playwright test

This will make the pipeline run fail if any of the playwright tests fails. If you also want to integrate the test results with Azure DevOps, use failOnStderr:false and the built-in PublishTestResults task like so:

jobs:
    - deployment: Run_E2E_Tests
      pool:
        vmImage: ubuntu-20.04
      container: mcr.microsoft.com/playwright:v1.27.1-focal
      environment: testing
      strategy:
        runOnce:
          deploy:
            steps:
            - checkout: self
            - task: Bash@3
              displayName: 'Run Playwright tests'
              inputs:
                workingDirectory: 'my-e2e-tests'
                targetType: 'inline'
                failOnStderr: false
                env:
                  CI: true
                script: |
                  npm ci
                  npx playwright test
                  exit 0
            - task: PublishTestResults@2
              displayName: 'Publish test results'
              inputs:
                searchFolder: 'my-e2e-tests/test-results'
                testResultsFormat: 'JUnit'
                testResultsFiles: 'e2e-junit-results.xml' 
                mergeTestResults: true
                failTaskOnFailedTests: true
                testRunTitle: 'My End-To-End Tests'

Note: The JUnit reporter needs to be configured accordingly via

in playwright.config.ts.

CircleCI​

Running Playwright on Circle CI is very similar to running on GitHub Actions. In order to specify the pre-built Playwright Docker image , simply modify the agent definition with docker: in your config like so:

executors:
   pw-focal-development:
     docker:
       - image: mcr.microsoft.com/playwright:v1.27.1-focal
   environment:
     NODE_ENV: development # Needed if playwright is in `devDependencies`

Note: When using the docker agent definition, you are specifying the resource class of where playwright runs to the 'medium' tier here. The default behavior of Playwright is to set the number of workers to the detected core count (2 in the case of the medium tier). Overriding the number of workers to greater than this number will cause unnecessary timeouts and failures.

Similarly, If you’re using Playwright through Jest, then you may encounter an error spawning child processes:

[00:00.0]  jest args: --e2e --spec --max-workers=36
Error: spawn ENOMEM
   at ChildProcess.spawn (internal/child_process.js:394:11)

This is likely caused by Jest autodetecting the number of processes on the entire machine (36) rather than the number allowed to your container (2). To fix this, set jest --maxWorkers=2 in your test command.

Sharding in Circle CI​

Sharding in Circle CI is indexed with 0 which means that you will need to override the default parallelism ENV VARS. The following example demonstrates how to run Playwright with a Circle CI Parallelism of 4 by adding 1 to the CIRCLE_NODE_INDEX to pass into the --shard cli arg.

  playwright-job-name:
    executor: pw-focal-development
    parallelism: 4
    steps:
      - run: SHARD="$((${CIRCLE_NODE_INDEX}+1))"; npx playwright test -- --shard=${SHARD}/${CIRCLE_NODE_TOTAL}

Jenkins​

Jenkins supports Docker agents for pipelines. Use the Playwright Docker image to run tests on Jenkins.

pipeline {
   agent { docker { image 'mcr.microsoft.com/playwright:v1.27.1-focal' } }
   stages {
      stage('e2e-tests') {
         steps {
            // Depends on your language / test framework
            sh 'npm install'
            sh 'npx playwright test'
         }
      }
   }
}

Bitbucket Pipelines​

Bitbucket Pipelines can use public Docker images as build environments. To run Playwright tests on Bitbucket, use our public Docker image (see Dockerfile).

image: mcr.microsoft.com/playwright:v1.27.1-focal

GitLab CI​

To run Playwright tests on GitLab, use our public Docker image (see Dockerfile).

stages:
  - test

tests:
  stage: test
  image: mcr.microsoft.com/playwright:v1.27.1-focal
  script:
  ...

Caching browsers​

By default, Playwright downloads browser binaries when the Playwright NPM package is installed. The NPM packages have a postinstall hook that downloads the browser binaries. This behavior can be customized with environment variables.

Caching browsers on CI is strictly optional: The postinstall hooks should execute and download the browser binaries on every run.

Exception: node_modules are cached (Node-specific)​

Most CI providers cache the npm-cache directory (located at $HOME/.npm). If your CI pipelines caches the node_modules directory and you run npm install (instead of npm ci), the default configuration

will not work. This is because the npm install step will find the Playwright NPM package on disk and not execute the postinstall step.

Travis CI automatically caches node_modules if your repo does not have a package-lock.json file.

This behavior can be fixed with one of the following approaches:

  1. Move to caching $HOME/.npm or the npm-cache directory. (This is the default behavior in most CI providers.)
  2. Set PLAYWRIGHT_BROWSERS_PATH=0 as the environment variable before running npm install. This will download the browser binaries in the node_modules directory and cache them with the package code. See managing browser binaries.
  3. Use npm ci (instead of npm install) which forces a clean install: by removing the existing node_modules directory. See npm docs.
  4. Cache the browser binaries, with the steps below.

Directories to cache​

With the default behavior, Playwright downloads the browser binaries in the following directories:

  • %USERPROFILE%\AppData\Local\ms-playwright on Windows
  • ~/Library/Caches/ms-playwright on MacOS
  • ~/.cache/ms-playwright on Linux

To cache the browser downloads between CI runs, cache this location in your CI configuration, against a hash of the Playwright version.

Debugging browser launches​

Playwright supports the DEBUG environment variable to output debug logs during execution. Setting it to pw:browser* is helpful while debugging Error: Failed to launch browser errors.

DEBUG=pw:browser* pytest

Running headed​

By default, Playwright launches browsers in headless mode. This can be changed by passing a flag when the browser is launched.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
   # Works across chromium, firefox and webkit
   browser = p.chromium.launch(headless=False)

On Linux agents, headed execution requires Xvfb to be installed. Our Docker image and GitHub Action have Xvfb pre-installed. To run browsers in headed mode with Xvfb, add xvfb-run before the Node.js command.

xvfb-run python test.py