dima/jung
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
149 commits 1 branch 0 tags 9 GiB
Commit graph

36 commits

Author SHA1 Message Date
Dmitry Kokorin
b47ff7234c sec07 stage 0: normalized RAW (parseable by xref), verbatim to data/isap-raw, INDEX.md 27 items 
- RAW re-scraped from official ISAP PDF (2025 ed., 2pp): multi-author items combined
  (Solms/Panksepp, Alcaro/Carta, Panksepp/Biven), same-author repeats for separate works
  (Kast x3, Krieger x3)
- xref.py AUTHOR_LINE: initials blob now handles 'C. A.'-style multi-initials
  (08-12: +1 recovered item each)
- MASTER-LIST sec07: 24 NEW (16 books + 8 articles) + 3 xref (CW8->sec01#03,
  Jacobi->sec01#20, Kast Dynamics->sec04#11)
 2026-09-26 11:45:14 +03:00
Dmitry Kokorin
f5cb34195e content audit sec01-06: fixes (audit_content.py + 12 file/card corrections) 
- tools/audit_content.py: per-file vs item-title triage (SUSPECT/SMALL/NO-TEXT/NO-CYR)
- sec01: .part removed; 10 files restored .pdf ext (LFS rename); #04/#19/#27 verify notes
- sec02 #16: corrupted double-encoded txt -> clean flib fb2 b/844509; #17/#22 doc verified via catdoc
- sec03 #11: -b epub = Spanish von Franz (Paidós 1983, forged EN OPF) -> bonus label
- sec03 #42: КАРО 2012 'Irish Tales' = EN reader (not RU, not the listed book) -> bonus rename
- sec03 #56: Onians b/c/d/e = Cambridge 24-87KB previews -> labeled
- sec06 #18: filename 1981->1989 (Perera 'Descent to the Goddess' 1989; 1981 = other book)
- sec06 #28: ia Zimmer OCR layer = foreign Devanagari text; images = RKP (p.100 verified)
- sec04 #30: buksmart 2020 = RU/FR bilingual w/ VeryPDF watermarks -> note
- linter: azw3/mobi ext + underscore in fname regex; manifest cross-check ignores off-disk rows;
  8 legacy cards section-order fixed (CR before DL)
- AGENTS.md: 'Content audit (2026-09-25)' lessons section
- check-md: 0/219
 2026-09-26 01:32:02 +03:00
Dmitry Kokorin
7d3794560e cross-section dedup system (user tasks 2+3, 2026-09-25): 
- tools/xref.py: master book list for ALL 12 sections (RAW 07-12 scraped: sections/07/RAW.md
  + data/isap-raw/), cross-ref matcher (CW gate + distinctive tokens + years + Cyrillic
  transliteration), stale MANIFEST 'Not downloadable' checker; data/MASTER-LIST.md generated
  (17 book-dupes in 01-06 found, e.g. sec01#51=sec06#14 Solomon)
- file-level dedup: 17 md5-identical files removed from later sections (kept in first),
  MANIFESTs/cards -> cross-ref rows, totals updated (sec01 102 / 02 37 / 03 125 / 04 38 /
  05 38 / 06 37), dedup rule in AGENTS.md + MANIFEST Notes
- sec05 MANIFEST phantom line '03 Aion' removed (Aion not in sec05 list; files in sec01)
- sec03 #44: 1968 Bodley file was #41's book (content 'English Fairy Tales') — removed,
  #44 keeps ABC-CLIO 2002
 2026-09-25 23:25:05 +03:00
Dmitry Kokorin
4274488410 authors: wiki-anchor audit — 1 wrong QID fixed (Homer: Q327970=Uzhevych → Q6691), 
9 wikipedia URLs normalized to wikidata, tools/audit_anchors.py (all-language label match)
 2026-09-25 22:39:01 +03:00
Dmitry Kokorin
b7ddfac17e sec06: key-vs-f_id discovery — 12 EN files recovered (6 rescues) + precheck v2 
- USER FINDING: libgen edition files map {key: {f_id, md5}} — KEY and f_id VALUE
  are DIFFERENT file records; we were downloading/prechecking by the KEY (wrong file).
- Recovered by f_id: 09-t1 (Eliade UCP 1978), 13 Sungod (Cornell 2010), 14 Solomon
  (Karnac 2007), 17 God-Image (Inner Light 1992), 19 Hornung (UNC 1982), 20 Psyche in
  Scripture (1995), 23 Dionysos (Bollingen LXV/2 Princeton) + backups 12/16/24/25/26/27.
  All content-verified. '24 systemic mislinks' = our bug, not libgen.
- lg.py: objects[]=f,e,s,a,p,w + topics[] (file-level search; found Solomon md5) +
  md5 in output. lgdl.py: bymd5 subcommand.
- precheck v2: rev-value match = OK; locator words >=2 shared = OK; words present
  0 shared = BAD; hash/ISBN/empty locator = no signal (SUSPECT); digit-runs masked
  before tokenization (hex 'aaae' trap).
- MANIFEST/SUMMARY/cards/AGENTS.md updated. 34 files, 608 MB (28 backup still downloading).
 2026-09-25 10:04:01 +03:00
Dmitry Kokorin
7b24100450 sec06: 28 карточек (11✅/2🔶/15❌) + rulabel.py (ru labels как канал канонической кириллицы) 
- Кросс-рефы: 01→sec01#05, 02→sec02#06, 03→sec01#32, 05→sec03#09, 06→sec01#15, 11→sec03#20
- Новые ✅: Eliade HRI («История веры и религиозных идей» Акад.проект 2008-09),
  Otto «Священное» СПбГУ 2008, Kerényi «Дионис» ВРС 2007, MacCulloch Э/Эксмо 2018,
  Schimmel «Мир исламского мистицизма» Садра 2012, Scholem 4 изд., von Franz CW6
  («Видения Николая из Флюэ…» Акад.проект 2025, rescue через rutitles diff)
- tools/rulabel.py: Wikidata ru label + aliases = каноническая кириллица
  (VIAF/ISNI/GND/BnF с этой машины недоступны: 403/mёртвые API). Ловушки:
  Циммер не Зиммер, Корбен не Корбин, Шолем не Шулем.
- AGENTS.md: источник 12 дополнен (rulabel.py + статус authority-APIs)
 2026-09-24 11:26:20 +03:00
Dmitry Kokorin
bc1f91d9be authors: systemic identity audit — Wiki anchors + 9 date/kinship corrections 
User caught: Emma Jung dossier said '1877-1965, Jung's eldest daughter'
(actually 1882-1955, his WIFE, Q124210). Systemic response:
- New MANDATORY field **Wiki anchor:** (QID + URL) in dossier canon (AGENTS.md)
- tools/verify_dossiers.py: resumable Wikidata audit (bulk wbgetentities for
  anchored, search+CANDIDATE review for the rest, 429 backoff, TSV report)
- 82 dossiers now anchored; 76 marked no-confirmed-QID (WIKI-CHECK notes)
- Date/kinship corrections found by the audit:
  * Emma Jung: wife not daughter, 1882-1955 (Q124210)
  * Verena Kast: b. 1943 not 1951, father Walter Kast not 'Hans Kast' (Q2515260)
  * James Hillman: 1926 not 1941 (Q934774)
  * Jolande Jacobi: 1890-1973 not 1902-1980 (Q88022)
  * Maria Czaplicka: d. 1921 not 1939 (Q532804)
  * Henri Ellenberger: 1905-1993 not 1900-1987 (Q115111)
  * Edward F. Edinger: 1922-1998 not 1905-2001 (Q5819984, en.wiki)
  * Hans Dieckmann: d. 2007 not 2005 (Q59526701)
  * Heinrich Zimmer: 1890-1943, anchored to the Indologist Q215922
    (not the Celtist Q76954)
- Homonym traps documented in TSV notes (John Hill=botanist, Karl
  Koch=hacker, Barbara Meier=model, Andrew Samuels=soccer, etc.)
 2026-09-24 10:29:54 +03:00
Dmitry Kokorin
2a2699d61a sec05: EN downloads complete (30 files) + SUMMARY at-a-glance + lgdl queue infra 
- 30/31 EN candidates downloaded via libgen, all content-verified
  (magic+pages+first-page text); 08b 37-pp fragment discarded
- EN gaps: 04/05/15/16/23 not found (rare academic), 03/14 ia-restricted
- MANIFEST.md: EN sections per item + not-downloadable extended
- 26 cards: Downloads tables w/ EN rows + EN NOT FOUND notes (6 cards)
- SUMMARY.md created (canon v1): at-a-glance table 29 items, EN->RU authors
- tools/lgdl.py: speed-probe + CDN-edge re-draw (per-request get.php
  302s to different storage hosts), connect timeout, stall wipe on first stall
- AGENTS.md: Lesson 9 (EN-queue infra: ads.php throttle, watchdog,
  unbuffered logs, dual-queue trap, AA 403, i2p verdict)
 2026-09-24 00:04:13 +03:00
Dmitry Kokorin
67fac97559 tools/lgdl.py: precheck <f_id> <edition_id> [pages] — mislinked-file gate (reverse-map editions + locator batch path + size sanity; exit 0/1/2); dl with 4th arg auto-prechecks and aborts on RED. Tested: caught 3 bad files (Одержимый магией / Laso / Captain Action comic); OK path verified 2026-09-23 00:49:32 +03:00
Dmitry Kokorin
d01e323e75 liart.py: OPAC-Global protocol probed (direct.exe FindView/FindSize, guest session); SearXNG record-URL fallback works; direct mode blocked on search labels+iddb (admin-only InfoDB) — data/liart-params.json stub for browser read 2026-09-22 08:42:06 +03:00
Dmitry Kokorin
31cb0dea70 unification: dossiers → CANON v1 (104/142 changed: H1 [dates] incl. open-ended for living, Homonym warnings unified, RU-name sections → canonical field, Round logs → Notes, Dossier status dropped) 2026-09-21 23:59:49 +03:00
Dmitry Kokorin
680bf45ee1 unification: SUMMARY → CANON v1 (Final counts, at-a-glance renamed, Authors—EN→RU generated for sec02/03 from dossiers, legacy blocks → Notes ###; sec04 totals row added; 0 rows lost) 2026-09-21 23:56:54 +03:00
Dmitry Kokorin
62bb11a99d unification: INDEX → CANON v1 (| # | Group | EN title | Author(s) | Status | File |), 219 rows, 0 status mismatches; sec02 legacy triage table dropped (dup of main table + cards) 2026-09-21 23:48:53 +03:00
Dmitry Kokorin
3225039d88 unification: sec04 MANIFEST → CANON v1 (Red Book +4 files, queue-table sources resolved from f_ids, page-count pollution fixed); 4 sections' manifests unified; 136 card DL sources normalized (MANIFEST→—, ia prefix, lg f/ ids) 2026-09-21 23:47:05 +03:00
Dmitry Kokorin
b01f539194 unification: sec03 MANIFEST → CANON v1 (per-item tables; source ids unrecorded → '—' + sweep pointer) 2026-09-21 23:43:39 +03:00
Dmitry Kokorin
dab9e312d9 unification: sec02 MANIFEST → CANON v1 (per-item tables, cross-section block preserved) 2026-09-21 23:42:36 +03:00
Dmitry Kokorin
62b16fa4bf unification: sec01 MANIFEST → CANON v1 (per-item tables, Totals); +4 missing EN files (01/07 ia byte-verified, 32/49); 8 cards DL fixed (sources, missing rows) 2026-09-21 23:41:18 +03:00
Dmitry Kokorin
f3fb432bd9 v3 migration: linter tools/check-md.py (0/219 issues); sec01 #47 RU table fixed; AGENTS.md: migration done, liart TODO + pending unification markers 2026-09-21 11:11:53 +03:00
Dmitry Kokorin
4f5372ab56 v3 migration: sec04 #22 — ES section + Original field (Spanish original) 2026-09-21 11:09:57 +03:00
Dmitry Kokorin
cde7a4cd4e sec03: Phase 0 — RAW + 68 card skeletons + INDEX + AUTHORS-EN (51 rows) + 39 new author dossiers + input.tsv + oa-sec03.json (68/68 crossref) 2026-09-19 14:40:19 +03:00
Dmitry Kokorin
34c6372836 sec02: Phase 0 — 57 book cards + RAW + INDEX + AUTHORS-EN + 33 author dossiers (full EN names resolved: Bosnak, Dieckmann=Hans, Alex=William, Hill=Clara, Mattoon, Marjasch, Shafton, Strauch/Meier, Solms, Roesler, Vedfelt, Kelsey, Delaney, Krippner, Aizenstat, Covitz, Lambert, Sanford, Wilmer, Kramer, Domhoff, Barrett/McNamara, etc.) 2026-09-18 20:46:29 +03:00
Dmitry Kokorin
5441dddbb8 pipeline v3: RU-title candidates (rutitles+ru-dict+pymorphy3), batch sweep driver, OpenAlex/Crossref 
- tools/rutitles.py: EN->RU token translation (data/ru-dict.tsv, 540+ pairs, Kogito/Castalia
  conventions) + Jaccard diff against data/ru-titles.jsonl (692 titles: Kogito 518, Litres 24,
  OPP 55, flibusta a/5272+57639+118921+122883+193690+193689). Matching is lemmatized
  (pymorphy3) — case endings handled: 'великой матери' -> 'Великая мать' 1.25.
  'The Great Mother' -> 'Великая мать' 1.25 top hit; 'The Symbolic Quest' -> correct negative.
- tools/sweep.py: batch driver for Phases 1-2 (rsl+lg+alib+flib+cogito+SearXNG-OZON-snippet
  parse+rutitles diff per item; --nlr optional). Query log data/queries.log (JSONL).
  Fixed: cogito nav-menu leak (parse bx_product_item only), libgen robot-block (lg.py curl fallback).
- tools/oa.py: OpenAlex + Crossref (Phase 0 identity/ISBN, no key; found Margaret Wilkinson,
  Karen Evers-Fahey, Symbolic Quest Princeton ISBN).
- data/sweeps/01-fundamentals/input.tsv: 16 ❌ items loaded; background sweep running.
- AGENTS.md: source 10b (oa.py), flibusta .su = reduced mirror (dropped from pipeline),
  pipeline v3 section (rutitles/oa/sweep/pymorphy3 note: pymorphy2 broken on py3.12).
- data/ru-titles.jsonl committed as the RU market universe asset.
 2026-09-18 10:56:36 +03:00
Dmitry Kokorin
32e505b2f9 sec01: phase 4 downloads complete (50 files, 204MB) + item 12 EN preview + flib.py pagination 
- 45 RU + 5 EN + 1 preview downloaded (flibusta.is OPDS, libgen, archive.org)
- 10 flib pdf stubs replaced via libgen (18,22,28,29,53) or fb2 twins
- flib 'fb2+zip' files unpacked to plain fb2
- item 12 (Evers-Fahey): RG=Request-PDF; PagePlace 23pp preview saved+verified
- evers-fahey dossier enriched (ofj.org bio)
- flib.py: authorall (rel=next pagination) + author <page>
- AGENTS.md: flib pagination/quirks + PagePlace preview tip
- INDEX: phase 4 done line; 8 trade-only items listed in MANIFEST
 2026-09-17 10:41:43 +03:00
Dmitry Kokorin
4420aef7c3 sec01 phase 2 (round 1): 17/55 items resolved 
- new ✅: 03 (Структура и динамика психического, Когито), 10 (Эго и архетип, 2 изд.),
  13 (Лекции по юнговской типологии, b/509587), 29 (Юнговская карта души, Стайн Мюррей 2010)
- ❌ with evidence: 02 (CW7), 05 (Aion — 'Эйон' 0), 16 (Shadow and Self), 21 (Animus and Anima),
  38 (Woman in the Mirror), 47 (Way of All Women)
- flibusta.is OPDS sweep (tools/flib.py): 30+ author feeds checked; bonus books logged
  (Edinger x10, Hillman x8, Harding x4, Henderson x2, Douglas Cambridge Handbook, Jung-E Grail Legend)
- tools/ia.py (archive.org EN search/metadata/files); AGENTS.md sources 10+13 updated
 2026-09-17 00:09:03 +03:00
Dmitry Kokorin
868fa9268a tools/mgu.py: MGU Scientific Library search (GET-discovery, fields/methods/pagination, ISB dead, SER goldmine); AGENTS.md source 22 + Google low-volume warning 2026-09-16 09:59:32 +03:00
Dmitry Kokorin
4feff7a0f7 ISBN validation (58 ISBNs): tools/isbnval.py + data/isbn-validation.tsv; SUMMARY Table 4 + fixed 2 bad-check-digit typos and 2 publisher-prefix mismatches; AGENTS.md source 21 2026-09-16 00:01:13 +03:00
Dmitry Kokorin
8b632f1229 lgdl.py: stall detection + fresh-start wipe + speed guard; MANIFEST queue status 2026-09-15 13:23:18 +03:00
Dmitry Kokorin
40f174db76 downloads: item 05 EN (Abt Daimon 2005/2017) + item 24 (Franz EN 1974 + RU Касталия 2018 djvu); items 03 big facsimiles + 05 RU + 11 RU in background queue 2026-09-15 12:17:59 +03:00
Dmitry Kokorin
e41464c2b4 downloads: item 01 (Jung CW9/1) — 7 PDFs (5 EN Princeton/Routledge 2nd eds + 2 RU АСТ 2019/2020), all size-verified + content-verified; tools/lgdl.py downloader (ads.php card → get.php key → bytes, resume, backoff) 2026-09-15 10:18:30 +03:00
Dmitry Kokorin
6decbd3146 NLR sweep round 7: all section-04 surnames checked (tools/nlr.py, CRW+Primo); Cunningham 2018 identified; Dora Kalff confirmed absent from NLR (correct spelling Калфф); Schaverien 'Умирающий пациент' in NLR holdings 2026-09-09 12:38:57 +03:00
Dmitry Kokorin
6dc0738584 #38 FOUND: «Символизм цвета. Из архивов Эраноса 1972» (Клуб Касталия, ISBN 978-5-90811-054-9) — same Ottmann editor as Spring 2005; via castalia.ru sitemap sweep (Phase 2b) 2026-09-09 00:46:19 +03:00
Dmitry Kokorin
59fb6d3570 alib.ru source (tools/alib.py, CP1251 stem-search marketplace): Turner handbook = Дипак 2015 ISBN 978-5-98580-069-2 (RSL 7609331); Furth 2nd ed 2014; Neumann market round 4 (КДУ/Питер/Маниф/Касталия editions, Angelika Lev bio 2021); notes #16/#26/#29 updated 2026-09-09 00:19:52 +03:00
Dmitry Kokorin
c0a3ac0f2f EN-side research pillars: tools/ol.py (Open Library) + tools/wd.py (Wikidata), Google Books ISBN pattern, archive.org verification, HSE shop-directory + NikBook/Skifiya/beta2alpha sources, workflow v2 (full names first, reverse publisher sweep, community lists, content verification gate); full EN author names resolved for all 39 items (AUTHORS-EN.md); CONTEXT round 3 (Neumann cogito finds, 1959 disambiguation) 2026-09-09 00:10:25 +03:00
Dmitry Kokorin
cf85e3f1ad sec04: Jung CW9/1 — ✅ «Архетипы и коллективное бессознательное» АСТ 2019/2020, libgen 141693619 2026-09-08 23:06:59 +03:00
Dmitry Kokorin
a6c062344f AGENTS/CONTEXT: new sources (cogito person pages, livelib author pages), verified RU editions (Franz, Jaffé/Man&HisSymbols, Lüscher, Kandinsky, Kast, Schaverien) 2026-09-08 22:28:09 +03:00
Dmitry Kokorin
9b8cf58a4d Init: research structure, tools (lg.py, rsl.py, sx.sh), AGENTS.md, CONTEXT.md, section 04 skeleton (39 book notes) 2026-09-08 22:10:07 +03:00