Metadata-Version: 2.4
Name: bs4-token-ext
Version: 0.0.2
Summary: Add your description here
License: MIT
License-File: LICENSE
Keywords: beautifulsoup,llm,tiktoken,token,web-scraping
Requires-Python: >=3.12
Requires-Dist: beautifulsoup4>=4.14.2
Requires-Dist: tiktoken>=0.12.0
Description-Content-Type: text/markdown

# bs4-token-ext
Add token counting capabilities to BeautifulSoup tags for LLM applications.
---
LLM（大規模言語モデル）向けに、BeautifulSoup のタグにトークン数をカウントする機能を追加する。
## Usage
```python
from bs4_token_ext import TokenAwareBeautifulSoup

html = "<div><p>Hello, world!</p></div>"
soup = TokenAwareBeautifulSoup(html, 'html.parser')
div = soup.find('div')
p = soup.find('p')

print(p.token_count) 
print(p.token_count_with_html)  
print(div.token_count)  
print(div.token_count_with_html)  
```