liart.py: OPAC-Global protocol probed (direct.exe FindView/FindSize, guest session); SearXNG record-URL fallback works; direct mode blocked on search labels+iddb (admin-only InfoDB) — data/liart-params.json stub for browser read
This commit is contained in:
parent
92b9d589a2
commit
d01e323e75
3 changed files with 187 additions and 5 deletions
22
AGENTS.md
22
AGENTS.md
|
|
@ -110,9 +110,10 @@ sections/0N-<name>/ # one .md per BOOK (chapter-level list items are merged int
|
|||
`| RU title | EN original | publisher, year | ISBN |` → `## Homonym warnings` (ЕДИНОЕ название,
|
||||
legacy-варианты: «Homonym warning (date)», «Homonyms / notes», «Омонимы») → `## Notes` (датированные
|
||||
append; legacy «Round N» логи сюда). Запрещено: `**EN identity:**` отдельно, `**Dossier status:**`.
|
||||
- **TODO (later, after 4 sections in order):** `tools/liart.py` — search for opac.liart.ru (РГБИ):
|
||||
platform «Библика» (api.bibliosearch.ru/bs.min.js), search goes via JS `simpleSearch()`,
|
||||
endpoint not yet probed; until then — SearXNG `site:opac.liart.ru <author>` only.
|
||||
- **TODO (later, after 4 sections in order):** `tools/liart.py` — **HALF DONE 2026-09-21:**
|
||||
OPAC-Global protocol reverse-engineered (direct.exe/FindView, guest session), SearXNG
|
||||
record-URL fallback works; BLOCKED on search labels + iddb (server InfoDB, admin-only for
|
||||
GUEST) — needs a one-time browser read (see data/liart-params.json).
|
||||
- **Author dossiers** (`authors/<lastname>-<firstname>.md`) are the **CANONICAL** home for
|
||||
identity, shared across ALL sections (same author can appear in 01, 04, …). One dossier per
|
||||
PERSON, slug `<lastname>-<firstname>` (lowercase, hyphens). Hold: dates, RU name candidates
|
||||
|
|
@ -385,8 +386,19 @@ sections/0N-<name>/ # one .md per BOOK (chapter-level list items are merged int
|
|||
|
||||
### 26. book.ivran.ru + opac.liart.ru — small RU library OPACs, curl-able GOST records (2026-09-20)
|
||||
- `book.ivran.ru/book?id=<N>` — «Книги сотрудников ИВ РАН» DB: full GOST rows (title, editor,
|
||||
city/year/pages, ISBN). `opac.liart.ru/record/<db>/<hex>` — РГБИ (State Art Library) OPAC.
|
||||
Find via SearXNG `site:book.ivran.ru` / `site:opac.liart.ru <author>`.
|
||||
city/year/pages, ISBN). Find via SearXNG `site:book.ivran.ru`.
|
||||
- **opac.liart.ru (РГБИ) — platform switched to OPAC-Global (DITM-Global)** (2026-09-21, probed):
|
||||
- `/opacg/` guest session: POST `arg0=GUEST&arg1=GUESTE&TypeAccess=PayAccess` to
|
||||
`/cgiopac/opacg/opac.exe`; session `9096453`/`GUEST` (hardcoded in JS).
|
||||
- Search = POST `/cgiopac/opacg/direct.exe` FLAT fields: `_service=opacfindd.FindView`
|
||||
(or `FindSize`), `query/body="<label> <value>"`, `iddb=<N>`, `userId`, `session`,
|
||||
`_xsl=_version=2.7.0/1.1.0` — full protocol in `tools/liart.py` docstring.
|
||||
- **BLOCKER:** valid search labels + iddb come from server InfoDB (admin-only for GUEST;
|
||||
iddb 1–4 absent, 7+ exist). Record pages `/record/<db>/<hex>` ARE curl-able (GOST in <title>).
|
||||
SearXNG `site:opac.liart.ru` index is sparse (author queries 0, `site:.../record` 20).
|
||||
- **Tool: `tools/liart.py`** — default mode = SearXNG record-URL fallback (works today);
|
||||
`--direct` mode ready, needs `data/liart-params.json` filled from the browser
|
||||
(Поиск → Базовый → F12: `input id=basic_<label>` / `select id=field_1`). TODO remains.
|
||||
|
||||
### 27. maap.pro «Библиотека» — RU article translations of Jungian authors (2026-09-20, user-provided)
|
||||
- `https://www.maap.pro/biblioteka/stati/` — МОСКОВСКАЯ АССОЦИАЦИЯ АНАЛИТИЧЕСКОЙ ПСИХОЛОГИИ
|
||||
|
|
|
|||
6
data/liart-params.json
Normal file
6
data/liart-params.json
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"_comment": "FILL FROM BROWSER: open https://opac.liart.ru/opacg/ then Поиск then Базовый; F12 Elements: input id=basic_<label> and select id=field_1 options. iddb: try 7..20 (1-4 not for GUEST, 7+ exist).",
|
||||
"iddb": null,
|
||||
"labels": {"author": "TBD", "title": "TBD", "free": "TBD"},
|
||||
"probe_labels": []
|
||||
}
|
||||
164
tools/liart.py
Normal file
164
tools/liart.py
Normal file
|
|
@ -0,0 +1,164 @@
|
|||
#!/usr/bin/env python3
|
||||
"""РГБИ (State Art Library of Russia, opac.liart.ru) search.
|
||||
|
||||
Platform: OPAC-Global (DITM-Global), switched from «Библика» — probed 2026-09-21.
|
||||
|
||||
Protocol (reverse-engineered from /opacg/js):
|
||||
1. GET /opacg/ → frameset (login as guest auto)
|
||||
2. POST /cgiopac/opacg/opac.exe arg0=GUEST&arg1=GUESTE&TypeAccess=PayAccess
|
||||
→ guest session (numsean=9096453 hardcoded in JS, identif=GUEST)
|
||||
3. POST /cgiopac/opacg/direct.exe (FLAT form fields, not querylist):
|
||||
_service = opacfindd.FindView (results) | opacfindd.FindSize (count)
|
||||
_version = 2.7.0 (FindView) | 1.1.0 (FindSize)
|
||||
_xsl = /opacg/html/search/xsl/search_results.xsl | .../history_find_size.xsl
|
||||
_errorXsl = /opacg/html/common/xsl/error.xsl
|
||||
_wait = 6M
|
||||
query/body = "<label> <value>" (FindView) | query=... (FindSize)
|
||||
length=10 start=0
|
||||
userId=GUEST session=9096453
|
||||
iddb=<logical DB number> (1..4 not found for GUEST; 7+ exist)
|
||||
level[i] = SearchLevels (server-provided, unknown)
|
||||
Query syntax (search_term_handlers.js): label + ' ' + value, e.g. "AUT Нойманн".
|
||||
KNOWN BLOCKER: valid search LABELS come from server InfoDB (code XML), which is
|
||||
admin-only for GUEST (opacsettingd.* on opac.exe → error page). Labels are rendered
|
||||
into the search form by JS in a real browser. To configure: open
|
||||
https://opac.liart.ru/opacg/ → «Поиск» → «Базовый» and fill data/liart-params.json:
|
||||
{"iddb": N, "labels": {"author": "...", "title": "...", "free": "..."}}
|
||||
(values = the exact label tokens; in the browser: F12 → Elements → the
|
||||
<input id="basic_<label>"> / <select id="field_1"> <option value="...">).
|
||||
|
||||
Modes:
|
||||
liart.py --direct "Нойманн" [FIELD] direct.exe FindView (needs data/liart-params.json)
|
||||
liart.py "Нойманн" SearXNG site:opac.liart.ru fallback (works today)
|
||||
liart.py --label-probe "Нойманн" brute a label list against FindSize (debug)
|
||||
|
||||
Polite: ~3s between direct.exe calls. Output: parsed records (title/author/year) or raw XML.
|
||||
"""
|
||||
import json, os, re, subprocess, sys, time, html
|
||||
|
||||
BASE = 'https://opac.liart.ru'
|
||||
CFG = os.path.join(os.path.dirname(__file__), '..', 'data', 'liart-params.json')
|
||||
UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36'
|
||||
|
||||
def curl(url, data=None, cookie=None):
|
||||
cmd = ['curl', '-s', '--max-time', '45', '-A', UA]
|
||||
if cookie:
|
||||
cmd += ['-b', cookie, '-c', cookie]
|
||||
if data:
|
||||
cmd += ['-X', 'POST', url]
|
||||
for k, v in data.items():
|
||||
cmd += ['--data-urlencode', f'{k}={v}']
|
||||
else:
|
||||
cmd.append(url)
|
||||
r = subprocess.run(cmd, capture_output=True, text=False)
|
||||
return r.stdout.decode('utf-8', 'replace')
|
||||
|
||||
def session(cook='/tmp/liart_cookies.txt'):
|
||||
open(cook, 'w').close()
|
||||
curl(BASE + '/opacg/', cookie=cook)
|
||||
r = curl(BASE + '/cgiopac/opacg/opac.exe',
|
||||
{'arg0': 'GUEST', 'arg1': 'GUESTE', 'TypeAccess': 'PayAccess'}, cookie=cook)
|
||||
return cook
|
||||
|
||||
def load_cfg():
|
||||
if not os.path.exists(CFG):
|
||||
sys.exit(f'no {CFG} — fill it from the browser form (see module docstring)')
|
||||
return json.load(open(CFG))
|
||||
|
||||
def findview(cook, iddb, query, length=20, start=0, raw=False):
|
||||
r = curl(BASE + '/cgiopac/opacg/direct.exe', {
|
||||
'_xsl': '/opacg/html/search/xsl/search_results.xsl',
|
||||
'_errorXsl': '/opacg/html/common/xsl/error.xsl',
|
||||
'_version': '2.7.0',
|
||||
'_service': 'opacfindd.FindView',
|
||||
'length': str(length), 'start': str(start),
|
||||
'query/body': query,
|
||||
'userId': 'GUEST', 'session': '9096453',
|
||||
'iddb': str(iddb),
|
||||
}, cookie=cook)
|
||||
if raw:
|
||||
return r
|
||||
# parse: results are <document>... records with MARC-ish fields; try common shapes
|
||||
texts = re.findall(r'<text[^>]*>(.*?)</text>', r, re.S)
|
||||
titles = re.findall(r'<title[^>]*>(.*?)</title>', r, re.S)
|
||||
if '<error' in r:
|
||||
err = ' | '.join(re.findall(r'<entry>([^<]*)</entry>', r)[:3])
|
||||
return {'error': err.strip()}
|
||||
recs = re.findall(r'<record.*?>(.*?)</record>', r, re.S)
|
||||
out = []
|
||||
for rec in recs:
|
||||
row = {}
|
||||
for m in re.finditer(r'<field[^>]*name="([^"]*)"[^>]*>(.*?)</field>', rec, re.S):
|
||||
name, val = m.group(1), re.sub(r'<[^>]+>', ' ', m.group(2)).strip()
|
||||
row.setdefault(name, val)
|
||||
out.append(row)
|
||||
return {'count': len(out), 'records': out, 'titles': [html.unescape(t) for t in titles[:30]]}
|
||||
|
||||
def findsize(cook, iddb, query):
|
||||
r = curl(BASE + '/cgiopac/opacg/direct.exe', {
|
||||
'_xsl': '/opacg/html/search/xsl/history_find_size.xsl',
|
||||
'_version': '1.1.0',
|
||||
'_service': 'opacfindd.FindSize',
|
||||
'query': query,
|
||||
'userId': 'GUEST', 'session': '9096453',
|
||||
'iddb': str(iddb),
|
||||
}, cookie=cook)
|
||||
m = re.search(r'<total[^>]*>(\d+)</total>|<findsize[^>]*>\s*(\d+)', r)
|
||||
if m:
|
||||
return int(m.group(1) or m.group(2))
|
||||
err = ' | '.join(re.findall(r'<entry>([^<]*)</entry>', r)[:2])
|
||||
return {'error': err.strip()} if err else 0
|
||||
|
||||
def searxng(q):
|
||||
import urllib.parse
|
||||
url = 'http://localhost:8888/search?' + urllib.parse.urlencode(
|
||||
{'q': q, 'format': 'json', 'language': 'ru'})
|
||||
r = subprocess.run(['curl', '-s', '--max-time', '30', url], capture_output=True, text=True)
|
||||
return json.loads(r.stdout or '{}').get('results', [])
|
||||
|
||||
def searxng_fallback(q):
|
||||
# record pages are curl-able: /record/<db>/<hex> with GOST line in <title>.
|
||||
# Google index of the site is sparse — try variants.
|
||||
seen = set()
|
||||
for query in (f'site:opac.liart.ru/record {q}', f'site:opac.liart.ru {q}'):
|
||||
for r in searxng(query):
|
||||
u = r['url']
|
||||
if u in seen:
|
||||
continue
|
||||
seen.add(u)
|
||||
print(f"{r['title']}")
|
||||
print(f" {u}")
|
||||
c = (r.get('content') or '').replace('\n', ' ')
|
||||
if c:
|
||||
print(f' {c[:220]}')
|
||||
print()
|
||||
if not seen:
|
||||
print('(0 results — site: index is sparse; try the OPAC in browser, or another catalog)')
|
||||
|
||||
def main():
|
||||
args = sys.argv[1:]
|
||||
if not args:
|
||||
sys.exit(__doc__)
|
||||
if args[0] == '--direct':
|
||||
cfg = load_cfg()
|
||||
cook = session()
|
||||
q = args[1]
|
||||
field = args[2] if len(args) > 2 else 'free'
|
||||
label = cfg['labels'][field]
|
||||
res = findview(cook, cfg['iddb'], f'{label} {q}', raw='--raw' in args)
|
||||
if isinstance(res, dict) and 'error' in res:
|
||||
print('ERROR:', res['error'])
|
||||
else:
|
||||
print(json.dumps(res, ensure_ascii=False, indent=1)[:6000])
|
||||
elif args[0] == '--label-probe':
|
||||
cfg = load_cfg()
|
||||
cook = session()
|
||||
for lab in cfg.get('probe_labels', []):
|
||||
n = findsize(cook, cfg['iddb'], f'{lab} {args[1]}')
|
||||
print(f'{lab:20} -> {n}')
|
||||
time.sleep(3)
|
||||
else:
|
||||
searxng_fallback(' '.join(args))
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Loading…
Add table
Add a link
Reference in a new issue