Metadata-Version: 2.4
Name: better-dbx-exceptions
Version: 0.1.0
Summary: Richer exception rendering for Databricks notebooks
Project-URL: Repository, https://github.com/lukeSmth/better-dbx-exceptions
Author: Luke Smith
License-Expression: MPL-2.0
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: databricks-sdk
Requires-Dist: ipynbname
Description-Content-Type: text/markdown

# better-dbx-exceptions

Richer exception rendering for Databricks notebooks. 

Has it ever taken you unreasonably long to track down the source of a PySpark error? Natively, errors only show the materializing code (`df.collect()`, `display(df)`, etc.) followed by the PySpark _client_ stacktrace. With `better-dbx-exceptions`, PySpark DataFrame errors point to the source of the failing expression (`F.lit(1) / F.lit(0)`).  

```python
df = spark.range(1).select(F.lit(1) / F.lit(0))
```

```python
df.collect()
```

=> 

**Native:**

```
[DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary
set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22012
== DataFrame ==
"__truediv__" was called from , line 1 in cell [9]
File <command-5875041447221118>, line 1
----> 1 df.collect()
... <------------------------------------------------------ 6 frames of PySpark client internals
File /databricks/python/lib/python3.12/site-packages/pyspark/sql/connect/client/core.py:2458, in
SparkConnectClient._handle_rpc_error(self, rpc_error)
   2454             logger.debug(f"Received ErrorInfo: {info}")
   2456             self._handle_rpc_error_with_error_info(info, status.message, status_code)  # EDGE
-> 2458             raise convert_exception(
   2459                 info,
   2460                 status.message,
   2461                 self._fetch_enriched_error(info),
   2462                 self._display_server_stack_trace(),
   2463                 status_code,
   2464             ) from None
   2466     raise SparkConnectGrpcException(
   2467         message=status.message,
   2468         sql_state=ErrorCode.CLIENT_UNEXPECTED_MISSING_SQL_STATE,  # EDGE
   2469         grpc_status_code=status_code,
   2470     ) from None
   2471 else:
```

**better-dbx-exceptions:**

```
---------------------------------------------------------------------------
ArithmeticException                       Traceback (most recent call last)
Cell 2, In[2], line 1
  (https://dbc-3cffa466-bd7e.cloud.databricks.com/editor/notebooks/2473432265319792
   ?o=7474649366507251#command/5875041447221116)
----> 1 df = spark.range(1).select(F.lit(1) / F.lit(0))

Fragment: "__truediv__"
ArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL
instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

SQLSTATE: 22012

Client stacktrace:
... <------------------------------------------------------ Native PySpark client stacktrace kept
```

## Install

```
%pip install better-dbx-exceptions
```

```python
from better_dbx_exceptions import setup

setup()
```

- `setup()` is idempotent (last call wins).
- Uncaught `PySparkException`s render with enriched information.
- Other exceptions use native display.
- Use `render_plain(exc)` to enrich and display manually caught exceptions.
- `teardown()` restores native behavior.

## Imported Expression Demo

Failing expression defined in imported module:

```python
from exprs import app_version_array_expr  # noqa: E402

df = spark.createDataFrame(
    ["1.0.0", "1.0.1", "9999999999.9999999999.9999999999"],
    schema="app_version: string",
)

with_app_array_df = df.withColumn("app_version_array", app_version_array_expr())
```

=> 

**Native:**

```
[CAST_INVALID_INPUT] The value '9999999999' of the type "STRING" cannot be cast to "INT" because it is malformed.
Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return
NULL instead. SQLSTATE: 22018
== DataFrame ==
"cast" was called from , line 5 in cell [3]
File <command-6090513257545418>, line 1
----> 1 with_app_array_df.collect()
... <------------------------------------------------------ 7 frames of PySpark client internals
```

**better-dbx-exceptions:** (construction chain rendered)

```
---------------------------------------------------------------------------
NumberFormatException                     Traceback (most recent call last)
Cell 15, In[3], line 8
  (https://dbc-3cffa466-bd7e.cloud.databricks.com/editor/notebooks/2204455071033272
   ?o=7474649366507251#command/6090513257545415)
      5     schema="app_version: string",
      6 )
      7 
----> 8 with_app_array_df = df.withColumn("app_version_array", app_version_array_expr())

File "/Workspace/bin/better-dbx-exceptions/examples/exprs.py", line 5
      2 
      3 
      4 def app_version_array_expr():
----> 5     return F.split(F.col("app_version"), "\\.").cast("array<int>")

Fragment: "cast"
NumberFormatException: [CAST_INVALID_INPUT] The value '9999999999' of the type "STRING" cannot be cast to "INT"
because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate
malformed input and return NULL instead.

SQLSTATE: 22018

Client stacktrace:
... <------------------------------------------------------ Native PySpark client stacktrace kept
```

See [`artifacts/main` branch](https://github.com/lukeSmth/better-dbx-exceptions/tree/artifacts/main) for rendered example notebooks.

## What `setup()` installs

- Modified call-site capture with its own depth control (`n_frames`), so deep capture works on serverless and without admin allowlisting of `spark.sql.stackTracesInDataFrameContext`.
- Expanded `_with_origin` wrap of DF API for expressions that natively carry no DataFrame context (e.g. `Column.cast`). 
  - Disable with `expand_watch=False`.
- A display hook for uncaught `PySparkException`s only; prior custom handlers are preserved and delegated to for non-`PySparkException`s.
- Suppression of the `DataFrameQueryContextLogger` JSON log.
  - Keep the log with `quiet_qc_logger=False`.

## Supported environments

Requires Spark 4.0+ internals and Python 3.12+. This limits supported compute to Serverless V4+ and Databricks Runtime 17+. 

Tested on:

- Compute: Serverless V4 and V5
- Locale: hosted and local (VSCode + Databricks Connect)
- Object Type: notebooks and scripts  

Expected to work but not rigorously tested:

- Compute: Classic DBR 17+
- Locale: local notebooks using IDEs other than VSCode (though enrichment may degrade)

Known unsupported:

- Compute: Serverless V3 and below, DBRs 16 and below (these run Spark 3.5 and below)
- `%run`-composed notebooks: frames from the child notebook are unmapped
- Non-Databricks PySpark (plain OSS Spark 4): untested; resolution assumes a Databricks context

## License

MPL-2.0
