Metadata-Version: 2.4
Name: cs-bs4utils
Version: 20261029
Summary: Various utility functions and classes for working with the HTML soup from `beautifulsoup4`.
Keywords: python3
Author-email: Cameron Simpson <cs@cskk.id.au>
Description-Content-Type: text/markdown
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Requires-Dist: beautifulsoup4
Requires-Dist: cs.gimmicks>=20260311
Requires-Dist: cs.lex>=20261016
Requires-Dist: cs.pfx>=20260912
Requires-Dist: cs.printt>=20261016
Requires-Dist: icontract
Requires-Dist: lxml
Requires-Dist: typeguard
Project-URL: MonoRepo Commits, https://bitbucket.org/cameron_simpson/css/commits/branch/main
Project-URL: Monorepo Git Mirror, https://github.com/cameron-simpson/css
Project-URL: Monorepo Hg/Mercurial Mirror, https://hg.sr.ht/~cameron-simpson/css
Project-URL: Source, https://github.com/cameron-simpson/css/blob/main/lib/python/cs/bs4utils.py

Various utility functions and classes for working with the HTML soup from `beautifulsoup4`.



Short summary:


* `as_xml`: Transform `tag` into an `lxml` XML element.


* `child_tags`: A generator yielding the immediate child tags of `child` whose tag name is `child_name`. If `child_name` is `None`, yield all the immediate child tags, skipping things like strings and comments.


* `find_heading`: Find the nearest heading satisfying the test `filter(tag)`. Return the tag or `None` is one is not found. The default `filter` tests that the heading is not empty. This uses `find_up` to locate the tag.


* `find_up`: A generator yielding `(found,ref)` 2-tuples obtained by search the tag tree left and up from `tag` using `.previous_sibling` and `.parent`, matching tags where `test(found)` is true.


* `printt_soup`: Print the contents of the soup via `cs.lex.printt` using `tabulate_soup` to make the table.


* `Table`: A `Widget` subclass representing an HTML TABLE tag.


* `tabulate_soup`: Return a table describing `soup` for use with `cs.lex.printt`. Connect tags with their child tags using Unicode box characters.


* `Widget`: Base class for various "widget" HTML constructs, such as a TABLE, or in principle anything else regular on a page.

# Functions

## as_xml(tag: bs4.element.Tag, *, E=None)

Transform `tag` into an `lxml` XML element.

## child_tags(tag, child_name: str | None = None) -> Iterable[bs4.element.Tag]

A generator yielding the immediate child tags of `child`
whose tag name is `child_name`.
If `child_name` is `None`, yield all the immediate child tags,
skipping things like strings and comments.

## find_heading(tag, filter: Callable[[bs4.element.Tag], bool] = <function <lambda> at 0x112f09d00>) -> bs4.element.Tag | None

Find the nearest heading satisfying the test `filter(tag)`.
Return the tag or `None` is one is not found.
The default `filter` tests that the heading is not empty.
This uses `find_up` to locate the tag.

## find_up(tag, test: Union[str, Callable[[bs4.element.Tag], bool]], *, first=False) -> Generator[tuple[bs4.element.Tag, bs4.element.Tag] | tuple[None, None], NoneType, NoneType]

A generator yielding `(found,ref)` 2-tuples obtained by
search the tag tree left and up from `tag` using `.previous_sibling`
and `.parent`, matching tags where `test(found)` is true.

The `test` may be a tag `.name` value (a string) or a callable
to evaluate a ound tag.

If `first` is true (default `False`) then the search stops
after the first match. If there are no matches the tuple
`(None,None)` is returned (this does not happen if `first`
is false).

A primary use case for this is to find the heading tag for `tag`.

In the tuple, `found` is the matched tag. `ref` is the reference
tag, the later sibling of `found` where the search started
for that level; if `found` is at the same level as `tag` then
`ref` will be `tag`.

For example, to locate the level 2 heading governing a tag:

    (h2,_), *_ = find_up(tag,lambda found: found.name == 'h2')

or more concisely:

    (h2,_), *_ = find_up(tag, 'h2')

or even:

    (h2,_), = find_up(tag, 'h2',first=True)

Note that the first two will cause Python to raise an exception
if there are no matches, while the third will provide `h2`
as `None`.

## printt_soup(tag: bs4.element.Tag, *, with_ancestors=False, without_children=False, **printt_kw)

Print the contents of the soup via `cs.lex.printt`
using `tabulate_soup` to make the table.

## tabulate_soup(tag: bs4.element.Tag | bs4.element.NavigableString, *, with_ancestors=False, without_children=False) -> list[list[str, str] | tuple]

Return a table describing `soup` for use with `cs.lex.printt`.
Connect tags with their child tags using Unicode box characters.

Parameters:
* `tag`: the `Tag` or `NovigableString` to tabulate
* `with_ancestors`: if true (default `False`), prepend the
  ancestor tags of `tag`
* `without_children`: if true (default `False`), omit the
  children of a `Tag`

# Classes

## class Table(Widget)

A `Widget` subclass representing an HTML TABLE tag.

### `Table.__init__(self, tag)`

Scan the TABLE for the basic structures, used for the other properties etc later.

Note that if there was no TBODY, the immediate rows of the
TABLE are presented as though they were in a single TBODY.

### `Table.all_rows`

    <functools.cached_property object at 0x112f238d0>

### `Table.as_indexed_values(self, *, convert: Optional[Callable[[str, int, int, int, bs4.element.Tag], Any]] = None, convert_head_cell: Optional[Callable[[str, int, int, int, bs4.element.Tag], Any]] = None, convert_body_cell: Optional[Callable[[str, int, int, int, bs4.element.Tag], Any]] = None, convert_foot_cell: Optional[Callable[[str, int, int, int, bs4.element.Tag], Any]] = None, omit_header=False, omit_footer=False) -> list[list[tuple[str, int, int, int, bs4.element.Tag, typing.Any]]]`

Return the table contents as a list-of-lists of indexed cell values.
Each inner list contains the cell value records from a row.

this is an elaborate counterpart to the `as_lists` method.

Parameters:
* `convert`: the default cell conversion function
* `convert_head`: the header cell conversion function, default from `convert`
* `convert_body`: the body cell conversion function, default from `convert`
* `convert_foot`: the footer cell conversion function, default from `convert`
* `omit_header`: do not include rows from the `THEAD` section
* `omit_footer`: do not include rows from the `TFOOT` section

The conversion functions accept the following positional parameters:
* `section_type`: one of `"THEAD"`, `"TBODY"` or `"TFOOT"`
* `section_index`: the index of the section, 0 for the
  header or footer but there may be multiple `TBODY` sections
* `row_index`: the index of the row within the section
* `col_index`: the index of the column within the row
* `cell`: the `TD` or `TH` tag for the cell
The function should return the converted value of `cell`.
The default conversion function returns `cell`.

The row and column indices supplied to the conversion unction
are of the _resolved_ cells, after expansion via the `colspan`
or `rowspan` values.
For example, a row with 3 cells whose second cell had a
`colspan=2` would be a list of 4 cells, with the second
original cell referenced in the second and third items of
the list; it _will_ be the same tag instance.

Each cell instance is converted only once; the same cell
spanning multiple columns or rows will have the same value
instance in the result record.

The resulting list-of-lists contains value records, a 6-tuple
of `(section_type,section_index,row_index,col_index,cell,value)`.
Note that the `row_index` and `col_index` are those of the
top left index where the `cell` was first encountered for
cells spanning multiple columns or rows.

Examples:

Convert every numeric cell to its `float` value, leave other cells as their text.

    def as_float(section_type, section_index, row_index, column_index, cell):
        text = cell.get_text.strip()
        try:
            value = float(text)
        except ValueError:
            value = text
        return value

    values = T.as_indexed_values(convert=as_float)

Convert only the body cells, keep the headers as tags, omit the footer:

    values = T.as_indexed_values(convert_body_cell=as_float, omit_footer=True)

### `Table.as_lists(self, *, omit_header=False, omit_footer=False) -> list[list[bs4.element.Tag]]`

Return the table contents as a list-of-lists-of-tags;
each inner list is a row of tags.
The innermost elements are the TH or TD tags.
Note that cells spanning multiple columns or rows via their
`colspan` or `rowspan` are the same reference.

Parameters:
* `omit_header`: do not include rows from the `THEAD` section
* `omit_footer`: do not include rows from the `TFOOT` section

### `Table.body_rows`

    <functools.cached_property object at 0x112f233d0>

### `Table.cell_colspan(cell: bs4.element.Tag) -> int`

Compute the `colspan` value for a table cell.

### `Table.cell_rowspan(cell: bs4.element.Tag) -> int`

Compute the `rowspan` value for a table cell.

### `Table.foot_rows`

    <functools.cached_property object at 0x112f23650>

### `Table.head_rows`

    <functools.cached_property object at 0x112f23350>

### `Table.printt(self)`

Print the table text.

### `Table.row_cells(tr: bs4.element.Tag) -> list[bs4.element.Tag]`

Return a list of the cells (`TD` or `TH`) from a `TR` tag.
`colspan` is supported by referencing the same cell multiple times.
Only `TD` and `TH` tags which are immediate children of the `TR` are recognised.

### `Table.section_rows(section: bs4.element.Tag | None) -> list[list[bs4.element.Tag]]`

Return the rows from a table section such as `THEAD`, `TBODY`, or `TFOOT`.
`rowspan` is supported by referencing the same cell in lower rows.

### `Table.title`

    <functools.cached_property object at 0x112f23950>

## class Widget

Base class for various "widget" HTML constructs, such as a
TABLE, or in principle anything else regular on a page.

A `Widgwt` supplies:
- `__init__(tag)` to record the target BS4 tag, typically the
  top level tag encompassing the widget
- `find_all(soup)`: returning a list of the top level tags
  within the BS4 tag `soup`; the default method calls
  `soup.find_all()` with the lower case version of the class
  name via `soup.find_all()`
- `scan(soup)`: a factory method calling `cls(tag)` for every
  tag found by `find_all(soup)`

Everything else in a subclass supports whatever needs doing
with the widget; the `Table` class is an exemplar:
- its `__init__` method passes the tag to `super().__init__()`
  as normal, then finds a few top level things about the table
  - the caption, header, bodies, footer
- the default `find_all` is used because the lass name matches
  the HTML tag name
- everything else more complex is provided as methods or
  `@cached_property` properties, computed on demand

### `Widget.__init__(self, tag: bs4.element.Tag)`

Initialise this `Widget` by saving `tag` as `self.tag` and
then calling `self.scan()`.

### `Widget.find_all(soup) -> list[bs4.element.Tag]`

The default `find_all` finds tags from `soup` whose name matches the class name.

### `Widget.scan(soup) -> list[typing.Self]`

Return a list of all `Widget`s of this type found in `soup`.
