AGENTS.md: source 3/9 corrections — cogito product pages curl-able w/ full specs; chitai-gorod product pages need CRW; Ozon blocked via CRW too (FAB); vse-svobodny cookie

This commit is contained in:
Dmitry Kokorin 2026-09-19 14:14:13 +03:00
parent 8a1b4ef84f
commit f59c49fbde

View file

@ -74,7 +74,11 @@ sections/0N-<name>/ # one .md per BOOK (chapter-level list items are merged int
### 3. cogito-shop.com — current Russian trade (Jungian/psychoanalysis specialists)
- URL: `https://cogito-shop.com/search/?q=<query>` (param is **q**, not query)
- Matches author AND title. Good for "is it sellable now in RU".
- Matches author AND title (fuzzy — «исцеляющее сновидение» missed, «Майер» hit). Good for
"is it sellable now in RU".
- **Product pages work with plain curl** (2026-09-19): «Характеристики» block in HTML —
Автор / Издательство / Формат / Вес / Тип обложки / Кол-во стр / Год / ISBN / Код. Search URL
= `/search/?q=<author>`, product link = `/catalog/<cat>/<slug>/`.
### 4. SearXNG (local) — cross-verification
- Tool: `tools/sx.sh "query"` (JSON API at http://localhost:8888; the web_search tool blocks localhost)
@ -116,9 +120,15 @@ sections/0N-<name>/ # one .md per BOOK (chapter-level list items are merged int
«Аборты…» Т8/ЦГИ 2017 is in GBS but NOT in RSL). Query via SearXNG `site:gbs.spb.ru <author>`.
### 9. chitai-gorod.ru / ozon / labirint — trade retail pages
- chitai-gorod author pages list RU works (some noise); product pages give publisher/year/ISBN.
- labirint.ru/books/<id> server-rendered w/ full specs (publisher, year, pages, ISBN, translator).
- Ozon product pages: blocked from curl (redirect loop) — fetch_content also fails; use browser.
- chitai-gorod author pages list RU works (some noise). **Product pages: CRW works** (2026-09-19) —
«Характеристики» block: Год издания / Кол-во стр / Переводчик / Издательство / ISBN. Plain curl
gives nothing (JS). Find product IDs via SearXNG `site:chitai-gorod.ru "<RU title>"`.
- labirint.ru book pages work with plain curl — «Характеристики» block after <h1> (publisher,
year, pages, translator, ISBN). Labirint SEARCH is anti-bot (text= ignored).
- Ozon product pages: blocked from curl (redirect loop) AND via CRW (FAB challenge 2026-09-19);
fetch_content also fails. Use SearXNG OZON snippets instead.
- vse-svobodny.com — works with `Cookie: beget=begetok` (anti-bot challenge, 2026-09-19);
product pages carry publisher/year/pages.
#### 10. flibusta — big e-book library. .su = server-rendered search; **.is = OPDS + direct downloads (user fixed access, 2026-07-16)**
- **.is OPDS (preferred):** Tool `tools/flib.py`:
@ -221,7 +231,8 @@ sections/0N-<name>/ # one .md per BOOK (chapter-level list items are merged int
### 18. Local CRW renderer (fastcrw.com, Firecrawl-compatible, localhost:3000)
- Tool: `tools/crw.py <url> [--html] [--wait N ms]` — POST /v1/scrape {"url","formats":["markdown"]}
- Use for JS-heavy sites curl can't read (soznanie.ast-academy.ru festival site rendered fine).
- Limits: Ozon and chitai-gorod search are API/anti-bot protected — CRW returns nothing.
- Limits: Ozon (FAB challenge) and chitai-gorod SEARCH are API/anti-bot protected — CRW returns
nothing. chitai-gorod PRODUCT pages DO render via CRW (see source 9).
- Insales shops (castalia.ru) don't need it: product data is server-side JSON-LD (curl OK).
### 19. NLR / РНБ (National Library of Russia) via Primo (primo.nlr.ru) + CRW