dima/jung
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

pipeline v3: RU-title candidates (rutitles+ru-dict+pymorphy3), batch sweep driver, OpenAlex/Crossref

Browse source 
- tools/rutitles.py: EN->RU token translation (data/ru-dict.tsv, 540+ pairs, Kogito/Castalia
  conventions) + Jaccard diff against data/ru-titles.jsonl (692 titles: Kogito 518, Litres 24,
  OPP 55, flibusta a/5272+57639+118921+122883+193690+193689). Matching is lemmatized
  (pymorphy3) — case endings handled: 'великой матери' -> 'Великая мать' 1.25.
  'The Great Mother' -> 'Великая мать' 1.25 top hit; 'The Symbolic Quest' -> correct negative.
- tools/sweep.py: batch driver for Phases 1-2 (rsl+lg+alib+flib+cogito+SearXNG-OZON-snippet
  parse+rutitles diff per item; --nlr optional). Query log data/queries.log (JSONL).
  Fixed: cogito nav-menu leak (parse bx_product_item only), libgen robot-block (lg.py curl fallback).
- tools/oa.py: OpenAlex + Crossref (Phase 0 identity/ISBN, no key; found Margaret Wilkinson,
  Karen Evers-Fahey, Symbolic Quest Princeton ISBN).
- data/sweeps/01-fundamentals/input.tsv: 16 ❌ items loaded; background sweep running.
- AGENTS.md: source 10b (oa.py), flibusta .su = reduced mirror (dropped from pipeline),
  pipeline v3 section (rutitles/oa/sweep/pymorphy3 note: pymorphy2 broken on py3.12).
- data/ru-titles.jsonl committed as the RU market universe asset.
This commit is contained in:
Dmitry Kokorin 2026-09-18 10:56:36 +03:00
parent 2157d058df
commit 5441dddbb8
11 changed files with 2101 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

15
data/queries.log Normal file
View file

@ -0,0 +1,15 @@
{"ts": "2026-09-18T10:44:15", "item": "30", "src": "rsl", "q": "Уайтмонт", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:44:23", "item": "30", "src": "libgen", "q": "Уайтмонт", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:44:27", "item": "30", "src": "alib", "q": "Уайтмонт", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:44:32", "item": "30", "src": "flibusta", "q": "Уайтмонт", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:44:34", "item": "30", "src": "cogito", "q": "Уайтмонт", "hits": 68, "sum": ""}
{"ts": "2026-09-18T10:44:45", "item": "30", "src": "rsl", "q": "Уитмонт", "hits": 2, "sum": ""}
{"ts": "2026-09-18T10:44:54", "item": "30", "src": "libgen", "q": "Уитмонт", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:44:59", "item": "30", "src": "alib", "q": "Уитмонт", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:45:24", "item": "30", "src": "flibusta", "q": "Уитмонт", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:45:26", "item": "30", "src": "cogito", "q": "Уитмонт", "hits": 68, "sum": ""}
{"ts": "2026-09-18T10:45:30", "item": "30", "src": "searxng", "q": "символический поиск", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:45:34", "item": "30", "src": "alib-title", "q": "символический поиск", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:45:42", "item": "30", "src": "searxng", "q": "символическая погоня", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:45:46", "item": "30", "src": "alib-title", "q": "символическая погоня", "hits": 0, "sum": ""}
{"ts": "2026-09-18T10:45:46", "item": "30", "src": "rutitles", "q": "The Symbolic Quest", "hits": 0, "sum": ""}