AGENTS.md: sec05 lessons (user oracle table, RU-title non-calques, mandatory rutitles diff, libgen mislinked-file trap, collection TOC verify) + sources 29-30 (fantlab, predanie.ru/LJ) + negatives gate v2

This commit is contained in:
Dmitry Kokorin 2026-09-23 00:05:38 +03:00
parent 9b492e21c2
commit be9080d0a5

View file

@ -7,8 +7,9 @@ find the **Russian edition(s)**: correct Russian title, publisher, year, transla
libgen download page (if exists), and РГБ (RSL) bibliographic record (if exists). libgen download page (if exists), and РГБ (RSL) bibliographic record (if exists).
We work **section by section**. Sections **01–04 are COMPLETE** (01: 38✅/2🔶/15❌; 02: 23✅/5🔶/29❌; We work **section by section**. Sections **01–04 are COMPLETE** (01: 38✅/2🔶/15❌; 02: 23✅/5🔶/29❌;
03: 38✅/30❌; 04: 12✅/1🔶/26❌). Sections 05–07 not started; 08–12 pending scope decision. 03: 38✅/30❌; 04: 12✅/1🔶/26❌). **Section 05 COMPLETE (2026-09-22): 17✅/12❌** — research +
Sections 1–3, 5–7 are skeletons for later. RU downloads; 7 rescues came from user tips (37% of the initial ❌ list was wrong — see Lessons).
Section 06–07 pending; 08–12 scope decision pending. Sections 1–3, 5–7 are skeletons for later.
## Repository layout ## Repository layout
@ -413,6 +414,52 @@ sections/0N-<name>/ # one .md per BOOK (chapter-level list items are merged int
-260-1, ред. Клименко А., сост. И. Надь) = «Сборник статей по античной мифологии» — a -260-1, ред. Клименко А., сост. И. Надь) = «Сборник статей по античной мифологии» — a
compilation, NOT *Essays on a Science of Mythology* (item 03 stays ❌). compilation, NOT *Essays on a Science of Mythology* (item 03 stays ❌).
### 29. fantlab.ru — structured work pages for RU translations (added 2026-09-22, sec05)
- `fantlab.ru/work<id>` — server-side HTML, curl-able (CRW for the publications block). Blocks:
«Перевод на русский» (translator + year + N изд.), «Издания» (period/language/translator filters),
«Входит в» (anthologies — e.g. «Бригантина 72-73»). Find work id via SearXNG
`site:fantlab.ru "<RU title>"`.
- Value: **Soviet-era / adventure / ethnographic / sci-fi books** that RSL and libgen miss
(found: van der Post «Потерянный мир Калахари» 1973, пер. Л. Деревянкина — sec05 #18;
Bjerrre «Затерянный мир Калахари» Мысль 1964 = homonym trap). Use for pre-1991 items.
### 30. predanie.ru + livejournal — older-edition pages & community oracles (added 2026-09-22, sec05)
- `predanie.ru/book/<id>-<slug>/` — curl-able book pages for older RU editions (Ладомир etc.),
GOST-like specs. Find via SearXNG `site:predanie.ru "<RU title>"`.
- **livejournal.com** reviews («Книжная полка культуролога» etc.) — CRW-rendered; often cite
BOTH RU title variants of one book (Benedict «Модели культуры»/«Образцы культуры»). Good
oracle for title-variant discovery; not a bibliographic source.
- **jungnewyork.com** — EN publisher/institute pages (per-book: TOC, praise, EN metadata);
NOT a RU source (book_mi.shtml = Adams EN page).
## Lessons from sec05 (2026-09-22) — what made the 7 rescues possible / what to do next time
1. **User oracle table = highest-ROI step.** Initial ❌ list (19) was 37% wrong; user checked the
RU-title-guess table against OZON/labirint memory and rescued 7. **Convention: at section end,
present the ❌ table (author, EN title, guessed RU author/title, channels probed) to the user**
before finalizing MISSING.md.
2. **RU titles are NOT calques** — the actual titles found: «Очерки сравнительного РЕЛИГИОВЕДЕНИЯ»
(not «религии»), «Аспекты мифа» (translated from FRENCH *Aspects du mythe*, not from EN *Myth and
Reality*), «Модели культуры» (not «Узоры/Образцы культуры»), «Потерянный мир Калахари» (the
«Затерянный» variant is a DIFFERENT book — Bjerrre). Always probe 2–3 RU title variants;
for authors with FR/DE originals (Eliade, van Gennep) also try the FR/DE-based RU title.
3. **rutitles diff was SKIPPED for sec05 and would have caught 3/7 rescues** («Аспекты мифа»,
«Культурный комплекс», «На пороге инициации» were all in data/ru-titles.jsonl from the cogito
crawl). **Mandatory: `tools/rutitles.py both "<EN title>"` for every item before ❌** (added to
the negatives gate below).
4. **flib authorall for prolific RU authors** (Элиаде a/2943 = 49 books) caught «Тайные общества»
(sec05 #13) that libgen title search missed. Always run flib author sweep for authors with
>5 RU books.
5. **libgen mislinked-file trap**: edition 141685581 «На пороге инициации» carried an ePubLibre
Spanish Lermontov file. Before downloading, check the file `locator`/title via
`json.php?object=f&addkeys=*&ids=<fid>`; for recent books prefer flib (cleaner metadata).
6. **Collection-inclusion questions** (#27/#28): download the smallest fb2, parse the TOC
(`<section name="title">`), grep for the EN book title + RU calque. «Символ и ритуал» (Наука
1983) CONTAINS «Ритуальный процесс» (full text, core section) but NOT «Лес символов»
(citations only).
7. **flib pdf download**: `/b/<id>/pdf` may return an HTML stub; `/b/<id>/download` → 302 →
direct `static.flibusta.is:443/b.usr/<File>.<ext>` URL — curl that directly.
## Catalog caching convention ## Catalog caching convention
Processed/scraper-unfriendly catalogs are cached under `data/catalogs/` (committed, with source Processed/scraper-unfriendly catalogs are cached under `data/catalogs/` (committed, with source
@ -516,8 +563,14 @@ not downloadable). Legend for ISBN columns: **✅** = ISBN found; **⚠** = publ
with a one-line totals row (DL EN / DL RU file counts). Cross-check the DL columns against the with a one-line totals row (DL EN / DL RU file counts). Cross-check the DL columns against the
section's `downloads/<nn>-*/MANIFEST.md` so the two never disagree. section's `downloads/<nn>-*/MANIFEST.md` so the two never disagree.
Negatives gate (before ❌): author sweep (RSL + libgen + cogito + flibusta) + one title-level Negatives gate (before ❌) — **v2 (2026-09-22, after sec05 rescues)**:
SearXNG + livelib author page + one shop/retail check. Log every RU query in MISSING.md. author sweep (RSL + libgen + cogito + flibusta **authorall** for prolific RU authors) +
**`tools/rutitles.py both "<EN title>"` diff (MANDATORY)** + 2–3 RU title variants
(calque + SearXNG/web_search `"<EN title>" перевод/на русском` + FR/DE-original variant for
Eliade/van Gennep-type authors) + livelib author page + one shop/retail check (cogito person
page for identity) + **fantlab.ru for pre-1991/adventure/ethnographic** + predanie.ru for older
editions + **user oracle table at section end** (see Lessons). Log every RU query in MISSING.md.
Before downloading from libgen: verify file locator/title (mislinked-file trap).
Special cases: Special cases:
- **Jung's CW volumes:** translated and widely available — don't deep-search; note the RU - **Jung's CW volumes:** translated and widely available — don't deep-search; note the RU