sec05: EN downloads complete (30 files) + SUMMARY at-a-glance + lgdl queue infra

- 30/31 EN candidates downloaded via libgen, all content-verified
  (magic+pages+first-page text); 08b 37-pp fragment discarded
- EN gaps: 04/05/15/16/23 not found (rare academic), 03/14 ia-restricted
- MANIFEST.md: EN sections per item + not-downloadable extended
- 26 cards: Downloads tables w/ EN rows + EN NOT FOUND notes (6 cards)
- SUMMARY.md created (canon v1): at-a-glance table 29 items, EN->RU authors
- tools/lgdl.py: speed-probe + CDN-edge re-draw (per-request get.php
  302s to different storage hosts), connect timeout, stall wipe on first stall
- AGENTS.md: Lesson 9 (EN-queue infra: ads.php throttle, watchdog,
  unbuffered logs, dual-queue trap, AA 403, i2p verdict)
This commit is contained in:
Dmitry Kokorin 2026-09-24 00:04:13 +03:00
parent 67fac97559
commit 2a2699d61a
60 changed files with 411 additions and 37 deletions

View file

@ -6,10 +6,11 @@ For every book in the ISAP Zurich reading list (https://isapzurich.com/en/librar
find the **Russian edition(s)**: correct Russian title, publisher, year, translator, ISBN,
libgen download page (if exists), and РГБ (RSL) bibliographic record (if exists).
We work **section by section**. Sections **01–04 are COMPLETE** (01: 38✅/2🔶/15❌; 02: 23✅/5🔶/29❌;
03: 38✅/30❌; 04: 12✅/1🔶/26❌). **Section 05 COMPLETE (2026-09-22): 17✅/12❌** — research +
RU downloads; 7 rescues came from user tips (37% of the initial ❌ list was wrong — see Lessons).
Section 06–07 pending; 08–12 scope decision pending. Sections 1–3, 5–7 are skeletons for later.
We work **section by section**. Sections **01–05 are COMPLETE** (01: 38✅/2🔶/15❌; 02: 23✅/5🔶/29❌;
03: 38✅/30❌; 04: 12✅/1🔶/26❌; **05: 17✅/12❌**). Section 05: research + RU downloads
2026-09-22 (7 rescues from user tips — see Lessons), **EN downloads COMPLETE 2026-09-23**
(30 files, 362 MB; 5 EN gaps unavailable + 2 ia-restricted; SUMMARY.md at-a-glance table done).
Section 06–07 pending; 08–12 scope decision pending. Sections 6–7 are skeletons for later.
## Repository layout
@ -474,6 +475,17 @@ sections/0N-<name>/ # one .md per BOOK (chapter-level list items are merged int
edition), `locator` batch path, size sanity; `dl <f_id> <dir> <name> <edition_id>` runs it
automatically and aborts on BAD. Mislink = skip that f_id → next file of the same edition →
next candidate edition → flib; log rejected f_ids in MANIFEST notes.
9. **EN-queue infra (2026-09-23, sec05 EN sweep)**: the libgen bottleneck is (a) `ads.php`
card-page IP-throttling under sustained load — pause 30–60 min, do NOT hammer (deepens the
block); (b) wedged CDN edges — each `get.php` 302s to a different storage host, so
`lgdl.py` now speed-probes (6 s range fetch) and re-draws the link on slow edges. Queue
pattern that survived the day: `queue.sh` = per-file `[ -f out ] && skip || timeout 1500
python3 -u lgdl.py dl ... >> "$LOG" 2>&1` + a **watchdog** bash loop (restart the queue if it
dies without a DONE marker — skip-existing makes restarts safe). Traps hit: python stdout
block-buffers to a file (use `-u` or logs look dead); generated-script typos (`>> LOG` vs
`>> $LOG`) — always `bash -n` a generated queue; killing a worker does NOT kill the parent
queue bash (dual-queue downloads corrupt the cookie jar). AA (annas-archive.gl) = 403 bot-wall
for anon downloads from this box; i2p not worth the router setup.
## Catalog caching convention
@ -636,6 +648,10 @@ The validated approach for closing the EN download gap of a section:
Result sec01: 50 files from 58 missing (8 genuinely absent: 13, 16, 38, 42, 48, 50, 52, 54 +
643 MB 24a replaced by 1991 twin).
Result sec05 (2026-09-23): 30 files from 31 candidates (08b 37-pp fragment discarded); 5 truly
unavailable (04 Chiron 1993, 05 1981, 15 SUNY 1997, 16 SUNY 1997, 23 Southern Book 1988) + 2
ia-restricted (03, 14). Queue infra: watchdog + speed-probe + `timeout 1500` per file +
unbuffered logs; ads.php throttle → pause, don't hammer.
## Download phase conventions (user rule, 2026-07-18)