Metadata-Version: 2.4
Name: HtmlCrawler
Version: 1.1.4
Summary: HtmlCrawler saves one HTML page and its stylesheets, scripts, and images, and can crawl a same-site set.
Author-email: Wilgat Wong <wilgat.wong@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/cloudgen/HtmlCrawler
Project-URL: Repository, https://github.com/cloudgen/HtmlCrawler
Project-URL: Issues, https://github.com/cloudgen/HtmlCrawler/issues
Keywords: CLI,TUI,download,crawl,HTML
Requires-Python: !=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: ChronicleLogger>=1.3.1
Dynamic: license-file

# HtmlCrawler - Crawl HTML pages up to a depth

![Version](https://img.shields.io/badge/Version-1.1.4-blue?style=flat-square)
![License](https://img.shields.io/badge/License-MIT-green?style=flat-square)
[![CIAO](https://img.shields.io/badge/Philosophy-CIAO%20(Caution%20%E2%80%A2%20Intentional%20%E2%80%A2%20Anti--fragile%20%E2%80%A2%20Over--engineered)-purple.svg)](https://github.com/cloudgen/ciao)
[![Stars](https://img.shields.io/github/stars/cloudgen/HtmlCrawler?style=flat-square)](https://github.com/cloudgen/HtmlCrawler)
[![Python](https://img.shields.io/badge/Python-2.7%2B%20(3.x%20recommended)-blue?style=flat-square)]()
[![PyPI](https://img.shields.io/pypi/v/HtmlCrawler?style=flat-square)](https://pypi.org/project/HtmlCrawler/)

HtmlCrawler crawls HTML pages on one host up to a depth, and up to a page cap when you pass `--max-pages`, and saves each page with its stylesheets, scripts, images, and fonts. The typed verb `crawl` is that walk. The typed verb `download` saves the one URL it is given and does not walk. Menu row **1 download** lists pages saved earlier, or accepts a URL. Menu row **2 list downloaded** lists saved page names, then opens the chosen page in Visual Studio Code or the default browser. Crawl is not a menu row.

On a terminal, starting it with no arguments opens a text menu. The menu does not download a page by itself, and it does not call pip. It does not join media, call an external encoder, or list files in the folder.

## Features

- Text menu on a terminal: **download**, **list downloaded**, **serve**, **System Log**, **self-management**, **language**, and **Exit**
- The first row shows the current folder and a local clock (`HH:MM:SS`) when the row has room
- Numbered rows line up. A rounded input box sits along the bottom, as wide as the terminal, with the name and version on the status line under the box
- **download** (1) lists saved `page name: URL` pairs, or accepts a URL, a page name, and an HTML file name. Press Enter to keep the name in brackets. A URL that ends with `guide.html` offers `guide/guide.html`. A URL that ends with a folder name such as `xxx` offers `xxx/index.html`. The typed verb `download --url URL` does the same and does not draw the menu. `--name` and `--html` override those names. A later run keeps the stored file and fills in files that are still missing
- **list downloaded** (2) lists saved page names by number. After a page is chosen, **21 edit** opens the saved HTML file in Visual Studio Code when the `code` command or the Visual Studio Code desktop entry is installed, and **22 view** opens it in the default browser when one is set. That file defaults to `index.html`. A missing tool is left off that menu. **0** returns to the page list, then to the main menu
- **serve** (3) lists saved page names by number, then asks for a port. An empty port uses **5000**. The page folder is served at `http://127.0.0.1:<port>/` until Ctrl+C. `/` is the saved HTML file. The typed verb `serve --page-name PAGENAME` does the same without the menu. `--port` changes the port
- The page is `~/.app/HtmlCrawler/output/<pagename>/<html file>`. The HTML file defaults to `index.html`. Linked stylesheets go in `output/<pagename>/assets/css/`. Linked scripts go in `output/<pagename>/assets/js/`. Linked images go in `output/<pagename>/assets/img/`. Those three folders stay flat, with no subfolders. Font files sit in `assets/css/`. When that HTML file is already there, the page is not downloaded again. A different name moves the existing file. Missing stylesheets, scripts, and images are fetched. Files already saved are kept.
- **System Log** (4) opens view-log, clear-log, and log-folder. **self-management** (6) opens version, about, and the pip lifecycle rows. **language** (7) picks the menu language
- Console script `html-crawler` and module entry `python -m HtmlCrawler`
- Typed verb `crawl`. It needs `--url`, `--name`, and `--depth`. `--max-pages` is optional and caps the number of HTML pages. It stays on the start host, reads `robots.txt` before any HTML page, and waits 1 second between requests to that host. It is not a menu row. `--depth` and `--max-pages` are refused on `download`
- Typed verbs: `help`, `version`, `about`, `download`, `crawl`, `serve`, `self-install`, `version-check`, `self-update`, `self-uninstall`

## Advantages

1. **Two ways to download.** `html-crawler download --url URL --name PAGENAME` saves the page in the terminal and does not draw the menu. On the menu, row **1 download** lists saved pages by number, or accepts a URL. With no arguments on a terminal, the text menu opens and waits. With no terminal and no verb, the program prints help and returns 0. There is no `--json`.
2. **Built-in languages.** Row **7** lists thirteen languages: English, Simplified Chinese, Traditional Chinese, Spanish, Arabic, French, Portuguese, Russian, German, Japanese, Korean, Dutch, and Greek. The choice is saved in `~/.local/HtmlCrawler/language` for the next run. `HTMLCRAWLER_LANG` overrides that file for one process and does not write it. The download verb stays English. The note on row 1 follows the menu language.
3. **Lifecycle and diagnostics.** `version-check`, `self-update`, `self-install`, and `self-uninstall` are menu rows **63**–**66** and typed verbs. `self-uninstall` on the command line needs `--force`. They call pip and do not use root. **System Log** (**4**) views a log, clears a log, and shows the log folder. **about** (**62**) stays in English.

## Quick Installation

**Python dependency:** `ChronicleLogger>=1.3.1` (required). Pip installs it with HtmlCrawler. No external media tool is required.

### PyPI (registry)

```bash
pip install HtmlCrawler
```

This installs the console entry **`html-crawler`** and the package **`HtmlCrawler`**. Packaging name SSOT is `pyproject.toml` `[project].name` = `HtmlCrawler`. This tree is **1.1.4**. The PyPI badge above shows the live index.

### Local install (checkout)

From a checkout (editable / unreleased work):

```bash
git clone https://github.com/cloudgen/HtmlCrawler.git
cd HtmlCrawler
python3 -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -e .
```

### Main menu

After install, on a terminal, `html-crawler` with no arguments prints this screen. The verb is bold and the note after the colon is italic. `9` leaves. `0` on a submenu goes back. After a command finishes, the main menu is shown again. A number that is not on the list prints an error and lets you choose again.

Picture of the English menu. Current directory `/tmp/clips`. The rows are the text the menu paints. The frame is the 100-column box.

```text
$ html-crawler
Path: /tmp/clips                                                                                     17:13:44

1. **download**       : *download a page and its stylesheets, scripts, and images*
2. **list downloaded**: *show saved page names*
3. **serve**          : *serve a saved page on this computer*
4. **System Log**     : *view, clear, and the log folder*
6. **self-management**: *version, about, and pip lifecycle*
7. **language**       : *display language for this menu*
9. **Exit**           : *leave*




╭─────────────────────────────────────────────────────────────────────────────────────────────────╮
│ >                                                                                               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯
  HtmlCrawler 1.1.4  │  main menu  │  Up/Down  •  Enter
```

**System Log** (4). Numbers 44–49 are not printed.

```text
Path: /tmp/clips                                                                                     17:13:46

41. **view-log**  : *list a log file and show it*
42. **clear-log** : *empty one log file*
43. **log-folder**: *show the log folder*
 0. **Back**      : *return to the main menu*




╭─────────────────────────────────────────────────────────────────────────────────────────────────╮
│ >                                                                                               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯
  HtmlCrawler 1.1.4  │  System Log  │  Up/Down  •  Enter
```

**language** (7). The short on each language row is that language’s own name. Numbers 84–98 are not printed. `0` goes back and does not save.

```text
Path: /tmp/clips                                                                                     17:13:45

71. **English**   : *use English for this menu*
72. **简体中文**      : *use 简体中文 for this menu*
73. **繁體中文**      : *use 繁體中文 for this menu*
74. **Español**   : *use Español for this menu*
75. **العربية**   : *use العربية for this menu*
76. **Français**  : *use Français for this menu*
77. **Português** : *use Português for this menu*
78. **Русский**   : *use Русский for this menu*
79. **Deutsch**   : *use Deutsch for this menu*
80. **日本語**       : *use 日本語 for this menu*
81. **한국어**       : *use 한국어 for this menu*
82. **Nederlands**: *use Nederlands for this menu*
83. **Ελληνικά**  : *use Ελληνικά for this menu*
 0. **Back**      : *return to the main menu*

╭─────────────────────────────────────────────────────────────────────────────────────────────────╮
│ >                                                                                               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯
  HtmlCrawler 1.1.4  │  language  │  Up/Down  •  Enter
```

**self-management** (6). Numbers 67–70 are not printed.

```text
Path: /tmp/clips                                                                                     17:13:47

61. **version**       : *show the installed version*
62. **about**         : *version and this computer*
63. **version-check** : *compare this install with pip*
64. **self-update**   : *upgrade this package with pip*
65. **self-uninstall**: *remove this package with pip*
66. **self-install**  : *install this package with pip*
 0. **Back**          : *return to the main menu*

╭─────────────────────────────────────────────────────────────────────────────────────────────────╮
│ >                                                                                               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯
  HtmlCrawler 1.1.4  │  self-management  │  Up/Down  •  Enter
```

Choose a number, or type the command name in the box. The block caret appears in that box while it is focused.

## Usage

```bash
html-crawler
# or
python -m HtmlCrawler
```

On a terminal, that opens the main menu above. It does not download a page yet, and it does not call pip. With no terminal, the same command prints help and returns 0.

**download** (1) stays on the menu. It lists saved `page name: URL` rows. Type a number to download that pair again, or type an `http://` or `https://` URL, then a page name, then an HTML file name. Press Enter on either question to keep the name in brackets. The brackets show the saved name, or the name taken from the URL. The typed verb does not draw the menu:

```bash
html-crawler download --url https://example.com/guide.html
html-crawler download --url https://example.com/xxx
html-crawler download --url https://example.com/ --name example
html-crawler download --url https://example.com/guide.html --name docs --html page.html
```

`guide.html` writes `~/.app/HtmlCrawler/output/guide/guide.html`. A URL that ends with a folder name, such as `xxx` or `xxx/`, writes `output/xxx/index.html`. `--name example` writes `output/example/index.html` when the URL does not name a file. `--html page.html` writes that file name instead. Linked stylesheets go in `output/example/assets/css/`, linked scripts in `output/example/assets/js/`, and linked images in `output/example/assets/img/`. Each of those folders is a single layer. The saved page points at those local files. A `<base>` tag or an `xml:base` attribute is removed so local links stay on the saved page. A canonical link and HTML comments are removed too. The URL, the page name, and the HTML file name are remembered in `~/.app/HtmlCrawler/output/url-map.json`. Running the same command again leaves that HTML file in place and downloads only the stylesheets, scripts, and images that are not already in the page folder. A different `--html` name moves the existing file and does not fetch the page URL again.

**crawl** is a typed verb. It is not a menu row. It needs `--url`, `--name`, and `--depth`. `--max-pages` is optional. It does not draw the menu:

```bash
html-crawler crawl --url https://example.com/ --name example --depth 1 --max-pages 20
```

That stays on the start host, reads `robots.txt` before any HTML page, and waits 1 second between requests to that host. A start path of `/` writes `~/.app/HtmlCrawler/output/example/index.html` and remembers the site in `url-map.json`. Any other path writes `example/<path>/index.html`. `--depth` and `--max-pages` are refused on `download`. A missing `--url`, `--name`, or `--depth` exits non-zero and names the flag. An omitted `--max-pages` keeps walking until `--depth` or the queue ends.

**list downloaded** (2) lists those page names by number. Choose one, then **21 edit** opens the saved HTML file in Visual Studio Code when `code` is installed, and **22 view** opens it in the default browser when a default browser is set. If either tool is missing, that row is not shown. **0** goes back.

**serve** (3) lists those page names by number. Choose one, then type a port. Press Enter with an empty box to use **5000**. The server listens on `127.0.0.1` and stops on Ctrl+C. The menu then shows `Stopped <name>.` The typed verb does not draw the menu:

```bash
html-crawler serve --page-name example
html-crawler serve --page-name example --port 8080
```

That serves `~/.app/HtmlCrawler/output/example/` so `http://127.0.0.1:5000/` is the saved HTML file, `index.html` unless `--html` chose another name, and `assets/` stays beside it.

```bash
html-crawler help
html-crawler version
html-crawler about
html-crawler download --url https://example.com/ --name example
html-crawler crawl --url https://example.com/ --name example --depth 1 --max-pages 20
html-crawler serve --page-name example
html-crawler version-check
html-crawler self-update
html-crawler self-install
html-crawler self-uninstall --force
```

`version` prints `HtmlCrawler 1.1.4` and does not call pip. `about` shows one English page: the product identity, a host check of this computer, and a star box. It does not call pip. `download` saves the page and does not draw the menu. `help` prints usage.

`version-check` runs `python -m pip index versions HtmlCrawler`. `self-update` runs `python -m pip install --upgrade HtmlCrawler`. `self-install` runs `python -m pip install HtmlCrawler`. `self-uninstall` runs `python -m pip uninstall -y HtmlCrawler` and needs `--force` on the command line. Those pip verbs do not use sudo. Empty arguments do not install or update.

Exit codes: non-zero for an unknown verb, a failed download, a failed crawl, a failed serve, a pip failure, or a menu that cannot open. Menu **Exit** (9) returns 0. A saved page returns 0. Ctrl+C on `serve` returns 0.

Import for scripts (thin entry):

```python
from HtmlCrawler import main
# interactive session; a terminal opens the text menu
```

## Screenshots

Each heading is the file name. The paragraph is what that picture shows: the words on the screen and the characters in the input box. The package is **1.1.4**. These pictures were captured on 2026-10-31 from the running text menu in `/tmp/clips`. The status line says **1.1.4**. On each menu picture the input box shows `>` and no typed text, and **1 download** or the first row of that board is highlighted. The package index can fetch a picture only after that file is on the public `main` branch.

### `language-menu.png`

Row **7** has opened the language list. **71 English** is highlighted, with the note "use English for this menu." The other rows are **72 简体中文**, **73 繁體中文**, **74 Español**, **75 العربية**, **76 Français**, **77 Português**, **78 Русский**, **79 Deutsch**, **80 日本語**, **81 한국어**, **82 Nederlands**, and **83 Ελληνικά**. Each note says to use that language for this menu. **0 Back** says "return to the main menu." The path label is `Path`. The path is `/tmp/clips`. The clock is `17:13:45` on the right of that row. The status line says `HtmlCrawler 1.1.4` and `language`.

![Language list, 71 English highlighted](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/language-menu.png)

### `main-menu-en.png`

English main menu. There is no saved-language line above the box. The path label is `Path`. The path is `/tmp/clips`. The clock is `17:13:44` on the right of that row. **1 download** is highlighted: "download a page and its stylesheets, scripts, and images." Then **2 list downloaded** "show saved page names," **3 serve** "serve a saved page on this computer," **4 System Log** "view, clear, and the log folder," **6 self-management** "version, about, and pip lifecycle," **7 language** "display language for this menu," and **9 Exit** "leave." The status line says `HtmlCrawler 1.1.4` and `main menu`.

![English main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-en.png)

### `main-menu-zh-hans.png`

Simplified Chinese main menu. The line above the box says `菜单语言是简体中文`. The path label is `路径`. The path is `/tmp/clips`. The clock is `17:13:51` on the right of that row. **1 download** is highlighted and stays `download`: "下载页面、样式表和脚本." **2** stays `list downloaded`. **3** stays `serve`. **4** is `系统日志`. **6** is `自我管理`. **7** is `语言`. **9** is `离开`. The status line says `HtmlCrawler 1.1.4` and `主菜单`.

![Simplified Chinese main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-zh-hans.png)

### `main-menu-zh-hant.png`

Traditional Chinese main menu. The line above the box says `選單語言是繁體中文`. The path label is `路徑`. The path is `/tmp/clips`. The clock is `17:13:52` on the right of that row. **1 download** is highlighted and stays `download`: "下載頁面、樣式表和指令碼." **2** stays `list downloaded`. **3** stays `serve`. **4** is `系統日誌`. **6** is `自我管理`. **7** is `語言`. **9** is `離開`. The status line says `HtmlCrawler 1.1.4` and `主選單`.

![Traditional Chinese main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-zh-hant.png)

### `main-menu-es.png`

Spanish main menu. The line above the box says `El idioma del menú es español`. The path label is `Ruta`. The path is `/tmp/clips`. The clock is `17:13:54` on the right of that row. **1 download** is highlighted and stays `download`: "descarga una página, sus hojas de estilo y sus scripts." **2** stays `list downloaded`. **3** stays `serve`. **4** is `registro`. **6** is `autogestión`. **7** is `idioma`. **9** is `Salir`. The status line says `HtmlCrawler 1.1.4` and `menú principal`.

![Spanish main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-es.png)

### `main-menu-ar.png`

Arabic main menu. The line above the box says `لغة القائمة هي العربية`. The path label is `المسار`. The path is `/tmp/clips`. The clock is `17:13:56` on the right of that row. The numbers stay on the left. The row words are the Arabic menu strings. **1 download** is highlighted and stays `download`. **2** stays `list downloaded`. **3** stays `serve`. **4** is `سجل النظام`. **6** is `إدارة ذاتية`. **7** is `لغة`. **9** is `خروج`. The status line says `HtmlCrawler 1.1.4` and `القائمة الرئيسية`.

![Arabic main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-ar.png)

### `main-menu-fr.png`

French main menu. The line above the box says `La langue du menu est le français`. The path label is `Chemin`. The path is `/tmp/clips`. The clock is `17:13:57` on the right of that row. **1 download** is highlighted and stays `download`: "télécharger une page, ses feuilles de style et ses scripts." **2** stays `list downloaded`. **3** stays `serve`. **4** is `journal`. **6** is `autogestion`. **7** is `langue`. **9** is `Quitter`. The status line says `HtmlCrawler 1.1.4` and `menu principal`.

![French main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-fr.png)

### `main-menu-pt.png`

Portuguese main menu. The line above the box says `O idioma do menu é português`. The path label is `Caminho`. The path is `/tmp/clips`. The clock is `17:13:59` on the right of that row. **1 download** is highlighted and stays `download`: "transferir uma página, as folhas de estilo e os scripts." **2** stays `list downloaded`. **3** stays `serve`. **4** is `registo`. **6** is `autogestão`. **7** is `idioma`. **9** is `Sair`. The status line says `HtmlCrawler 1.1.4` and `menu principal`.

![Portuguese main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-pt.png)

### `main-menu-ru.png`

Russian main menu. The line above the box says `Язык меню — русский`. The path label is `Путь`. The path is `/tmp/clips`. The clock is `17:14:01` on the right of that row. **1 download** is highlighted and stays `download`: "скачать страницу, таблицы стилей и скрипты." **2** stays `list downloaded`. **3** stays `serve`. **4** is `системный журнал`. **6** is `самоуправление`. **7** is `язык`. **9** is `Выход`. The status line says `HtmlCrawler 1.1.4` and `главное меню`.

![Russian main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-ru.png)

### `main-menu-de.png`

German main menu. The line above the box says `Die Menüsprache ist Deutsch`. The path label is `Pfad`. The path is `/tmp/clips`. The clock is `17:14:02` on the right of that row. **1 download** is highlighted and stays `download`: "eine Seite, ihre Stylesheets und ihre Skripte herunterladen." **2** stays `list downloaded`. **3** stays `serve`. **4** is `Systemprotokoll`. **6** is `Selbstverwaltung`. **7** is `Sprache`. **9** is `Beenden`. The status line says `HtmlCrawler 1.1.4` and `Hauptmenü`.

![German main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-de.png)

### `main-menu-ja.png`

Japanese main menu. The line above the box says `メニューの言語は日本語`. The path label is `パス`. The path is `/tmp/clips`. The clock is `17:14:04` on the right of that row. **1 download** is highlighted and stays `download`: "ページとスタイルシートとスクリプトを取得する." **2** stays `list downloaded`. **3** stays `serve`. **4** is `システムログ`. **6** is `自己管理`. **7** is `言語`. **9** is `終了`. The status line says `HtmlCrawler 1.1.4` and `メインメニュー`.

![Japanese main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-ja.png)

### `main-menu-ko.png`

Korean main menu. The line above the box says `메뉴 언어는 한국어`. The path label is `경로`. The path is `/tmp/clips`. The clock is `17:14:06` on the right of that row. **1 download** is highlighted and stays `download`: "페이지와 스타일시트와 스크립트를 받습니다." **2** stays `list downloaded`. **3** stays `serve`. **4** is `시스템 로그`. **6** is `자기관리`. **7** is `언어`. **9** is `종료`. The status line says `HtmlCrawler 1.1.4` and `주 메뉴`.

![Korean main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-ko.png)

### `main-menu-nl.png`

Dutch main menu. The line above the box says `De menutaal is Nederlands`. The path label is `Pad`. The path is `/tmp/clips`. The clock is `17:14:07` on the right of that row. **1 download** is highlighted and stays `download`: "download een pagina, de stylesheets en de scripts." **2** stays `list downloaded`. **3** stays `serve`. **4** is `systeemlog`. **6** is `zelfbeheer`. **7** is `taal`. **9** is `Afsluiten`. The status line says `HtmlCrawler 1.1.4` and `hoofdmenu`.

![Dutch main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-nl.png)

### `main-menu-el.png`

Greek main menu. The line above the box says `Η γλώσσα του μενού είναι ελληνικά`. The path label is `Διαδρομή`. The path is `/tmp/clips`. The clock is `17:14:09` on the right of that row. **1 download** is highlighted and stays `download`: "κατέβασε μια σελίδα, τα φύλλα στυλ και τα σκριπτ." **2** stays `list downloaded`. **3** stays `serve`. **4** is `αρχείο καταγραφής`. **6** is `αυτοδιαχείριση`. **7** is `γλώσσα`. **9** is `Έξοδος`. The status line says `HtmlCrawler 1.1.4` and `κύριο μενού`.

![Greek main menu](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/main-menu-el.png)

### `self-management.png`

Row **6** has opened self-management. **61 version** is highlighted: "show the installed version." Then **62 about** "version and this computer", **63 version-check** "compare this install with pip", **64 self-update** "upgrade this package with pip", **65 self-uninstall** "remove this package with pip", **66 self-install** "install this package with pip", and **0 Back** "return to the main menu." The path label is `Path`. The path is `/tmp/clips`. The clock is `17:13:47` on the right of that row. The status line says `HtmlCrawler 1.1.4` and `self-management`.

![Self-management, 61 version highlighted](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/self-management.png)

### `tui-about.png`

**about** (62) on the result page. The title is `HtmlCrawler (1.1.4) — result`. The page prints `HtmlCrawler 1.1.4`, `Domain: Crawl a same-site set of HTML pages up to a depth and a page cap.`, `Runtime tools: none`, and `Entry points: html-crawler, python -m HtmlCrawler`. The host check is stamped `2026-10-31 17:13:49.368015` and headed `[CHECK SYSTEM]:`. Visible lines include Python 3.12.3, C Library GCC 13.3.0, Ubuntu 24.04.5 LTS, amd64, the current user, the shell `/bin/bash`, Python Executable `python3`, python2 location, python3 location, a blank conda location, pyenv location, `Inside docker container: False`, `Cython String: cpython-312-x86_64-linux-gnu`, `Binary Type: amd64-glibc`, the program location, a process id, the cache folder in use, the preferred cache folder, two cache fallbacks, persistence storage, and `TTY / Interactive: yes`. The footer says `Press a key to return to the main menu.` There is no input box and no clock. The page stays in English. The star box says `HtmlCrawler (1.1.4) by Wilgat Wong on 2026-10-04` and `UNINSTALLED`, then Basic Usage `html-crawler download --url URL --name PAGENAME` and the homepage `https://github.com/cloudgen/HtmlCrawler`.

![About host check, English](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/tui-about.png)

### `system-log.png`

Row **4** has opened System Log. **41 view-log** is highlighted: "list a log file and show it." Then **42 clear-log** "empty one log file", **43 log-folder** "show the log folder", and **0 Back** "return to the main menu." The path label is `Path`. The path is `/tmp/clips`. The clock is `17:13:46` on the right of that row. The status line says `HtmlCrawler 1.1.4` and `System Log`.

![System Log, 41 view-log highlighted](https://raw.githubusercontent.com/cloudgen/HtmlCrawler/main/screenshots/system-log.png)

## Examples

```bash
html-crawler
```

```bash
html-crawler version
```

`version` prints `HtmlCrawler 1.1.4` and does not call pip. A run that is not the text menu can also print ChronicleLogger status lines above that. The text menu keeps those lines off the screen.

```bash
html-crawler download --url https://example.com/ --name example
```

That saves the page and does not draw the menu. On a terminal, row **1** lists saved pages or accepts a URL.

```bash
html-crawler crawl --url https://example.com/ --name example --depth 1 --max-pages 20
```

That crawls the same host and does not draw the menu. Empty arguments do not crawl.

## Platform Compatibility

| Platform | Status |
|----------|--------|
| Linux | Primary; tested development path |
| macOS | Supported when CPython is on PATH |
| Windows | Supported when CPython is on PATH (venv activate differs) |
| Architectures | Any with CPython |

The text menu needs a terminal. With no terminal and no verb, the program prints help and returns 0. It does not wait.

## Related Projects

- [HtmlCrawler on GitHub](https://github.com/cloudgen/HtmlCrawler) — this program’s source
- [HtmlCrawler on PyPI](https://pypi.org/project/HtmlCrawler/) — this program on PyPI
- [AnimeDlp](https://github.com/Wilgat/AnimeDlp) — command-line downloader for anime video sites
- [ChronicleLogger](https://github.com/Wilgat/ChronicleLogger) — status logger this program depends on (`ChronicleLogger>=1.3.1`)
- [VideoSpeed](https://github.com/Wilgat/VideoSpeed) — cuts, changes speed, and boomerangs a clip. That program is not this one
- [CIAO](https://github.com/cloudgen/ciao) — Caution, Intentional, Anti-fragile, Over-engineered
- [CIAO-Lite](https://github.com/cloudgen/ciao-lite) — short agent contract
- [safe-rm](https://github.com/cloudgen/safe-rm) — guarded `rm`

## Contributing

1. Keep product law under `docs/requirements/` in sync when behavior changes.
2. Prefer small, CIAO-safe changes. Do not put media joining, an external encoder, or a file listing back without an explicit design.
3. Version dual SSOT: bump **`pyproject.toml`** and **`src/HtmlCrawler/__init__.__version__`** together.
4. Open issues and pull requests on GitHub.

## License

MIT — see [`LICENSE.md`](./LICENSE.md). Also declared in `pyproject.toml`.

## Last Update

2026-10-31 — **1.1.4**. `--max-pages` on `crawl` is optional. Omit it and the walk stops at the required `--depth`. The only pip dependency is `ChronicleLogger>=1.3.1`. Menu pictures were captured again on 2026-10-31. The status line says `HtmlCrawler 1.1.4`. The about picture stamp is `2026-10-31 17:13:49.368015`. The about picture domain line is `Crawl a same-site set of HTML pages up to a depth and a page cap.` The Version badge matches `pyproject.toml` and `__version__`.

2026-10-31 — **1.1.3**. The package name, console script, and public source are HtmlCrawler, `html-crawler`, and https://github.com/cloudgen/HtmlCrawler. The Version badge matches `pyproject.toml` and `__version__`. Menu pictures below were captured on 2026-10-28 and still show `HtmlCrawler 1.1.2` on the status line. This release did not open the text menu, so those pictures were not recaptured.

2026-10-28 — **1.1.2**. Public source is https://github.com/cloudgen/HtmlCrawler. The console script is `html-crawler`. `download` can omit `--name` when the URL ends with `.html`, `.htm`, or a folder name such as `xxx`. `--html` names the saved HTML file. `crawl --url URL --name SITENAME --depth N --max-pages N` saves HTML pages on that host under `~/.app/HtmlCrawler/output/`. Crawl is not a menu row. Menu pictures show this board: language **71**–**83**, one main menu for each language, self-management **61**–**66**, about, and System Log **41**–**43**. The about picture prints the crawl domain sentence. Version badge matches `pyproject.toml` and `__version__`. `ChronicleLogger>=1.3.1` is required.
