Metadata-Version: 2.4
Name: selenium-crawler
Version: 0.1.3
Summary: Creates a simple Selenium Webdriver, optimized for web scraping.
Home-page: https://github.com/jo-hoe/selenium-crawler
Author: Jo-Hoe
License: MIT
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: selenium>=4.0.0
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Selenium Crawler

[![Test](https://github.com/jo-hoe/selenium-crawler/actions/workflows/test.yml/badge.svg)](https://github.com/jo-hoe/selenium-crawler/actions/workflows/test.yml)
[![Release](https://github.com/jo-hoe/selenium-crawler/actions/workflows/release.yml/badge.svg)](https://github.com/jo-hoe/selenium-crawler/actions/workflows/release.yml)

A simple, opinionated Selenium WebDriver setup optimized for web scraping with Chrome.
Designed for quick, lightweight data extraction, especially in Dockerized environments.

The project is not affiliated with Selenium in any way, except for using it as a dependency.

## Features

- Chrome WebDriver configuration for scraping
- Headless mode and image loading control
- Optimizations for Docker, Windows, Linux, and MacOS
- Utilities for element selection and waiting

## Requirements

- Google Chrome browser
- ChromeDriver (compatible with your Chrome version)

## Usage

Import and use the WebDriver creator and scraping tools in your scripts:

```python
from selenium_crawler.webdrivercreator import create_webdriver
from selenium_crawler.scraping_tools import get_element
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException

driver = create_webdriver(headless=True)
try:
    driver.get('https://example.com')
    element = get_element(driver, (By.XPATH, '//h1'), timeout=10)
    print(element.text)
except TimeoutException:
    print('Element not found within the timeout period.')
finally:
    driver.quit()
```

## Docker

To run successfully in Docker, use the following images and settings.

```Dockerfile
FROM python:3.13-alpine3.20

## Install system dependencies and update certificates
RUN apk update && apk upgrade --no-cache && \
    apk add --no-cache chromium-chromedriver ca-certificates && \
    update-ca-certificates && \
    rm -rf /var/cache/apk/*

# Set display port and dbus env to avoid hanging
ENV DISPLAY=:99
ENV DBUS_SESSION_BUS_ADDRESS=/dev/null

...
```

Other images and settings may work, but have not been tested.

## Testing

Run all tests with:

```sh
make test
```

Or manually:

```sh
python -m unittest discover -s tests -v
```

## Notes

- This project is not intended to bypass all bot detection mechanisms.
- Focuses on ease of use and quick setup for scraping tasks.
- For advanced anti-bot evasion, consider additional tools or services.
