Metadata-Version: 2.4
Name: pytest-runtime-xfail
Version: 1.1.1
Summary: Call runtime_xfail() to mark running test as xfail.
Project-URL: Home, https://github.com/okken/pytest-runtime-xfail
Author: Brian Okken
License: MIT License
        
        Copyright (c) 2021 Brian Okken
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE.txt
Classifier: Development Status :: 5 - Production/Stable
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Utilities
Requires-Python: >=3.8
Requires-Dist: pytest>=5.0.0
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: tox; extra == 'dev'
Requires-Dist: tox-uv; extra == 'dev'
Description-Content-Type: text/markdown

# pytest-runtime-xfail

pytest plugin, providing a `runtime_xfail` fixture, which is callable as `runtime_xfail()`, to allow runtime decisions to mark a test as `xfail`. 

## Installation

Install with pip:

    pip install pytest-runtime-xfail

## Usage

Include the fixture, then call it if you want to mark a test as `xfail` during runtime.

```python
 def test_something(runtime_xfail):
     if (runtime_condition):
        runtime_xfail()
     # ... the rest of your test
```

Can also be used in a fixture, of course.

```python
@pytest.fixture()
def foo(runtime_xfail):
  if (runtime_condition):
     runtime_xfail()
  # ... the rest of your fixture

def test_something(foo):
  # ... the rest of your test
```

## Reason this plugin is needed

pytest allows you to mark tests as expected to fail, or xfail, in two ways.

1. `@pytest.mark.xfail`. This allows you to mark tests or test parametrizations as `xfail` during test collection time.
   * pytest runs tests marked with `xfail` just like any other test.
   * If the test fails, it will result in `XFAIL`.
   * If it passes, `XPASS`. Unless you have `xfail_strict=true` or `@pytest.mark.xfail(strict=True)`, in which case, passing xfail-marked tests will result in `FAIL`.
      * This is useful to be alerted when an expected failing test starts to pass.

2. `pytest.xfail()`. If you need information only known at runtime to decide if `xfail` is appropriate, you can call `pytest.xfail()` during a test or fixture.
   * pytest runs the test as normal UNTIL `pytest.xfail()` is called.
   * When `pytest.xfail()` is called, the test execution stops and the test results in `XFAIL`.
   * The rest of the test is not run.
   * There is no way to get `XPASS` from `pytest.xfail()`.
   * `xfail_strict` has no effect.


There are times when we want a combination of these behaviors.

* We don't know until runtime if we should mark a test as `xfail`.
* We want the test run.
* We want the possibility of both `XFAIL` and `XPASS` results.
* We want to be able to use `xfail_strict=true` to alert us when the test starts passing.

This plugin fills that gap.

## Alternatives

You can get around the same limitation yourself by adding the marker through the `requests` object:

```python

def test_something(request):
     if (runtime_condition): 
        request.node.add_marker(pytest.mark.xfail(reason='some reason'))
     # ... rest of test
```

That's basically what this plugin does, just in a fixture.


## Example found in example/test_xfail.py

```python
"""
Run this with
* pytest -v
* pytest -v -o xfail_strict=true
"""

import pytest

@pytest.mark.xfail()
def test_marker_pass():
    'Can be XPASS or FAIL (if xfail_strict)'
    assert True

@pytest.mark.xfail()
def test_marker_fail():
    'Will always be XFAIL'
    assert False  # this statememt will be run

def test_old_xfail_pass():
    'Will always be XFAIL'
    pytest.xfail()
    assert True  # this statememt will NOT be run

def test_old_xfail_fail():
    'Will always be XFAIL'
    pytest.xfail()
    assert False  # this statememt will NOT be run

def test_runtime_xfail_pass(runtime_xfail):
    runtime_xfail()
    assert True  # this statement will be run

def test_runtime_xfail_fail(runtime_xfail):
    runtime_xfail()
    assert False  # this statement will be run

def test_runtime_xfail_reason(runtime_xfail):
    runtime_xfail(reason="for demo")
    assert False  # this statement will be run
```

**Output:**

```
(venv) $ pytest -v test_xfail.py 
========================= test session starts ==========================
collected 7 items                                                      

test_xfail.py::test_marker_pass XPASS                            [ 14%]
test_xfail.py::test_marker_fail XFAIL                            [ 28%]
test_xfail.py::test_old_xfail_pass XFAIL                         [ 42%]
test_xfail.py::test_old_xfail_fail XFAIL                         [ 57%]
test_xfail.py::test_runtime_xfail_pass XPASS                     [ 71%]
test_xfail.py::test_runtime_xfail_fail XFAIL                     [ 85%]
test_xfail.py::test_runtime_xfail_reason XFAIL (for demo)        [100%]

==================== 5 xfailed, 2 xpassed in 0.05s =====================
(venv) $ pytest -v test_xfail.py -o xfail_strict=true
========================= test session starts ==========================
collected 7 items                                                      

test_xfail.py::test_marker_pass FAILED                           [ 14%]
test_xfail.py::test_marker_fail XFAIL                            [ 28%]
test_xfail.py::test_old_xfail_pass XFAIL                         [ 42%]
test_xfail.py::test_old_xfail_fail XFAIL                         [ 57%]
test_xfail.py::test_runtime_xfail_pass FAILED                    [ 71%]
test_xfail.py::test_runtime_xfail_fail XFAIL                     [ 85%]
test_xfail.py::test_runtime_xfail_reason XFAIL (for demo)        [100%]

===================== 2 failed, 5 xfailed in 0.04s =====================
```