). Fetch the data files directly. - Start here: fetch /data/index.json (self-describing master index with schemas, URL patterns, freshness). - All data files are plain Parquet/JSON under /data/... — anonymous for GDELT/laws/derived-stats (raw market OHLCV needs an API key — see below), CORS *, HTTP Range OK. If you cannot reach / (e.g. a bot-management challenge on the HTML page), the /data/* file endpoints are the access path. ## Periodic polling (recommended procedure for recurring agents) 1. GET https://azsh.uk/status.json (<1 KB) — freshness of all datasets + pipeline health + anomaly flags. If nothing is stale and anomaly.flags is empty, you are done. 2. GET https://azsh.uk/signals.json — cross-domain signals (GDELT x market x laws) with analysis axes. Each signal has suggested_queries: the exact endpoints to fetch for the actual analysis. 3. If GDELT changed: GET /data/gdelt/events?since= for new batches, or /v1/query for aggregates. Send If-None-Match with the previous ETag to get 304 when unchanged. 4. If a signal is interesting: follow its suggested_queries, then form your own analysis. This service presents signals + analysis axes; the actual analysis is yours. - All /data/* endpoints support ETag/304. Polling without changes costs a few bytes. ## Signal detection (cross-domain analysis axes) - /signals.json — every 15 min: GDELT country/topic/lane surges (7d baseline), volume spikes, tone drops, JPX17 sector movers, laws pool updates, and cross-domain geo-market compounds. (A deeper daily cross-domain report with CAMEO action-type backing is at /data/laws/signals/daily/YYYYMMDD.json, daily 06:30 UTC.) - Each signal: {type, domain, severity, key, desc, suggested_queries[]}. severity: high|medium|low. - Cross-domain signals (e.g. cross_geo_market) flag where GDELT surges coincide with market moves — correlation, not causation; confirm with your own analysis. - /status.json.anomaly — hourly Z-score anomaly (event volume spike, tone drop) with baseline stats. ## Time-series query (server-side aggregation, <=64 KB responses) - GET https://azsh.uk/v1/query?from=YYYYMMDD&to=YYYYMMDD&agg=timeline&bucket=1h — event volume/tone per hour (or 1d) - GET /v1/query?agg=top_events&lane=main&actor1=US&max_rows=20 — top events with SOURCEURL + Page_Title (original article links) - GET /v1/query?agg=groupby_country | groupby_lane | groupby_topic — distributions - Filters: lane (main|JP|TCK), topic (economy|diplomacy|military|energy|industry|japan_economy|other), actor1/actor2 (ISO country), q (text on title/desc/actors) - Free tier: max 7-day range. Full 13-month history is parquet (/data/gdelt/parquet/). ## Japanese laws (pre-legislation government intent) - Bill/regulation pool with LLM-extracted rationale (why the government is acting, target sector/JSIC): https://azsh.uk/data/laws/laws/laws_meta.parquet (pool size: https://azsh.uk/data/laws/_manifests/laws_meta.json — DuckDB: SELECT * FROM 'https://azsh.uk/data/laws/laws/laws_meta.parquet') - CAMEO-coded law analysis: https://azsh.uk/data/laws/analysis/laws_cameo_summary.json - **As-Of query** (which pool laws were in force on a date): https://azsh.uk/data/laws/asof?date=YYYYMMDD§or=FINANCE_FINTECH — SCD2 有効期間から時点再現 (valid_from<=date&from=YYYY-MM-DD&until=YYYY-MM-DD&recordPacking=json (full text since 1947, 100 records/req, no CORS — proxy if needed; meeting list: /api/meeting, full meeting text: /api/meeting_list) ## Analysis foundation (frozen market + GDELT + laws — reproducible research layer) - **Period brief (start here for "what happened in [D1,D2]")**: GET https://azsh.uk/data/analysis/manifest?from=YYYYMMDD&to=YYYYMMDD (compact, <32 KB — data versions + per-file sha256 + caveats) → then per-day L0 summaries https://azsh.uk/data/analysis/{YYYYMMDD}.json (~2 KB each, anonymous). Full file list + sha256: &full=1. - **Universe (public, no prices)**: https://azsh.uk/data/analysis/universe.json — ticker + name + sector17 + has_daily/has_15m flags (~3.9k tickers) for selectors. - **Analysis recipes (deterministic procedures, versioned)**: https://azsh.uk/recipes (JSON index with recipe_version) — /recipes/news-market-reaction.md (event-study: event_session, ret_*, ar_*, date-cluster tests), /recipes/period-brief.md (standard [D1,D2] review), /recipes/law-market-impact.md (law event window study), /recipes/chart-analysis.md (shared chart reference: same payload sha256 + through). - **Charts (shared reference point, key required — per-ticker OHLCV)**: https://azsh.uk/api/market/chart?symbol=7203.T&resolution=1d (15m also; from/to slicing; through= pins the exact payload version — same through = same sha256 = same payload). 404 no_chart = not charted; 503 not_ready = backend pending. Human-facing page: https://azsh.uk/chart (fixture demo without a key). - Provenance: **observed** (frozen at observation time — preferred) vs **backfilled** ("today's view" — yfinance retroactive adjustments). L0 is published same-day; the cross-domain (reactions) block completes ~3 days later. Record manifest hash + recipe_version + code git hash with every analysis. ## Machine Index - https://azsh.uk/data/index.json — self-describing master index (schemas, URL patterns, freshness, caveats). **Read this first.** - https://azsh.uk/ai/docs.md — data dictionary for AI agents (column semantics, lane/topic values, tone definitions, what NOT to do) - https://azsh.uk/signals.md — human/agent-readable signal report (same data as signals.json, markdown) - https://azsh.uk/data — landing page (HTML, no JS required) - https://azsh.uk/data/cameo.json — CAMEO 2.0 event code reference (the parquet column EventCodeDescription is misnamed and holds the numeric CAMEO base code; use this for descriptions) ## Quality & reconciliation (trust) - https://azsh.uk/data/quality.json — permanent machine-readable quality status: per-dataset freshness/coverage/reconciliation + integrity method. **Check this for data trust.** - https://azsh.uk/quality — human-readable viewer of the above - https://azsh.uk/data/quality/sha256/{YYYYMMDD}.json — per-date sha256 manifest of GDELT parquet files (verify with: curl -s | sha256sum) - https://azsh.uk/data/quality/gdelt_recon_summary.json — rolling 90-day reconciliation summary: pipeline (curated) vs GDELT 2.0 source, subset property (no fabricated events), retention rate - https://azsh.uk/data/quality/recon_gdelt_YYYYMMDD.json — per-day detail (diff samples: what curation excluded) - Curated pipeline is a strict subset of GDELT source (verified weekly; subset_pass=true = zero fabricated/altered events). Low retention_pct is expected (noise exclusion). ## GDELT 15-minute events (curated, ~1.6k rows/day, live with 15-30 min lag) - File list: https://azsh.uk/data/gdelt/files?date=YYYYMMDD - Download (parquet, Range OK): https://azsh.uk/data/gdelt/parquet/YYYYMMDDHHMMSS.parquet — **storage tiers**: R2 keeps a 90-day hot window (true 15-min files); dates older than that transparently fall back to the HF cold tier (one daily parquet per day, same schema, 15-min rows preserved — response header X-Content-Granularity: daily). Direct cold access: https://azsh.uk/data/gdelt/hf/YYYYMMDD.parquet · file list + per-file sha256: https://azsh.uk/data/gdelt/hf/files · source: https://huggingface.co/datasets/Toshi33/azsh-uk-data (gdelt/daily/) - **Long-horizon pre-aggregates (hourly x country x topic x lane x CAMEO root, 13 months, ~30KB/day)**: https://azsh.uk/data/gdelt/agg/YYYYMMDD.parquet (file list: https://azsh.uk/data/gdelt/agg/files). tone_sum = SUM(AvgTone), same semantics as /v1/query. Load all files into DuckDB/Polars for SQL analysis. - **Latest batch as plain JSON (no DuckDB/pyarrow required)** — for agents that cannot fetch or parse parquet: https://azsh.uk/data/gdelt/events/latest.json (lite view: top 30 rows by NumMentions + key columns, ~12KB, fits 32KB fetch limits). Full batch: /data/gdelt/events/latest_full.json · specific batch: /data/gdelt/events/YYYYMMDDHHMMSS.json (full rows/columns). - Daily summary: https://azsh.uk/data/gdelt/summary/YYYYMMDD.json (or .../latest.json) - Key columns: EventCode (CAMEO 2.0), GoldsteinScale (-10..+10, cooperation/conflict strength — NOT a sentiment score), AvgTone (article-level tone, the event row's tone column), V2Tone (GKG 7-value comma list: tone,positive,negative,polarity,activity,self-reference,word_count — FIRST value is the article tone ~-100..+100; do NOT average the list, last value is word count), Actor1Name/Actor2Name, Actor1/Actor2CountryCode (ISO 3166-1 alpha-3: CHN, JPN), ActionGeo_CountryCode (FIPS 10-4: CH, JA — different system from actor codes), Lane (main=global|JP=Japan-related|TCK=CN/KR/TW-related), Topic (economy|diplomacy|military|energy|industry|japan_economy|other), Page_Title, SOURCEURL, Datetime - Lane/Topic are ETL classifications (human-validated R=91.3%) ## Japanese market ### Access (market gate — raw OHLCV requires an API key) - **API key required** (raw OHLCV, yfinance/Stooq source — not licensed for redistribution): 15m/*, freq=daily/* (incl. per-ticker JSON), quarterly/* - Send header 'X-API-Key: ' or query '?api_key=' (note: '?key=' is the data-key parameter, not auth) - No registration — keys are issued by the operator (request via the site operator). Without a key these endpoints return 401. - **No auth required** (public-statistics redistribution + own classifications/summaries): intl_flows/* (13F=SEC EDGAR, COT=CFTC, jpx_weekly=JPX), baskets/*, _config/*, manifests, list - Anonymous derived data is LIMITED TO NON-REVERSIBLE SECTOR-LEVEL AGGREGATES (yfinance terms target personal use; derived-data treatment unconfirmed). Per-ticker price/return/volatility series stay key-gated. - Freshness manifest: https://azsh.uk/data/market/manifests (check last_written=false = not yet written) - 15m bars (parquet, **key required**): https://azsh.uk/data/market/file?key=15m/YYYYMMDDHHMMSS.parquet - **Latest 15m bars as plain JSON (no DuckDB/pyarrow required)** — for agents that cannot fetch or parse parquet: https://azsh.uk/data/market/file?key=15m/latest.json (specific bar: key=15m/YYYYMMDDHHMMSS.json). Envelope {"ts": "...", "bars": [{ticker, open, high, low, close, volume}]} — ts shared at top level, prices rounded to 4 dp, 232 instruments per bar (~20 KB, fits a 32 KB fetch cap). - Daily bars + technicals (parquet, **key required**): key=freq=daily/year=YYYY/month=MM/market_YYYYMM.parquet (whole month in one file) - **Per-ticker snapshot (latest trading day, ~0.5-1 KB, key required)** — the primary path for agents fetching individual instruments: https://azsh.uk/data/market/file?key=freq=daily/ticker/{TICKER}.json — one row, all 24 columns (price + ma_50d/ma_200d/volatility/beta + instrument master). Example: key=freq=daily/ticker/6758.T.json (SOFTBANK). Updated daily after the Tokyo close. - Latest trading day, all ~3,900 tickers as plain JSON (key required): https://azsh.uk/data/market/file?key=freq=daily/latest.json (~2.3 MB — use the per-ticker endpoint above if your fetch is size-capped) - Quarterly fundamentals (key required): key=quarterly/TICKER_YYYYQN.parquet (parquet-only) - International flows, no auth (public statistics: SEC EDGAR / CFTC / JPX): 13F key=intl_flows/13f/quarter={YYYYQn}/13f_japan_{YYYYQn}.parquet (per-ticker: .../13f_japan_by_ticker_{YYYYQn}.parquet) · COT key=intl_flows/cot/year={YYYY}/cot_nikkei_{YYYY}.parquet · JPX weekly key=intl_flows/jpx_weekly/year={YYYY}/jpx_{YYYY}.parquet — list with: https://azsh.uk/data/market/list?prefix=intl_flows/ - **Disclosure archives (EDINET filings, no auth, permanent mirror — original URLs expire in ~31 days)**: https://azsh.uk/data/disclosures/{docID}.zip (120/160 earnings flash XBRL→CSV) · /{docID}/csv.zip (event filings 130/150/170/180/190/220/230/350/360) · /{docID}/att/{file} (PDF attachments). These are the evidence_url targets of the market signal datasets — fetch them to verify any signal. - **TDnet: NOT redistributed here (since 2026-10-15)** — JPX terms: viewing service serves only 31 days; older filings are paid (TDnet DBS/API); third-party redistribution may incur per-use fees. TDnet evidence_url in signal datasets points to the JPX original (https://www.release.tdnet.info/inbs/{disc_num}.pdf, expires ~31 days). /data/disclosures/tdnet/* returns 410. Inquiry: api-service@jpx.co.jp. - File list: https://azsh.uk/data/market/list?prefix=15m/ ## What does not exist (do not try these) - Market frequencies: **15m and daily only**. There is NO hourly / 5min / 1min / weekly / monthly / yearly data. Other key prefixes return 400 "key not allowed" by design — stop retrying. - No SQL API, no per-ticker history JSON (history = monthly parquet or Hugging Face), no registration/account system (auth = static API key for raw OHLCV only, see Market access section). (There IS a GDELT aggregation query API: /v1/query — see above.) - GDELT: 15-minute batches only. No hourly/daily parquet (use /v1/query or the daily summary JSON for aggregates). - No full-text search over the 13-month GDELT history (7-day query window + parquet for the rest). ## Payload sizes (plan fetches if your tool caps responses, e.g. 32 KB) - GDELT events/latest.json: ~12 KB (lite view: top 30 rows by NumMentions + key columns — fits 32 KB-cap tools; full batch at /data/gdelt/events/latest_full.json) - freq=daily/ticker/{TICKER}.json: ~0.5-1 KB (single instrument, key required) - 15m/latest.json: ~20 KB (all 232 instruments, newest bar, key required) - freq=daily/latest.json: ~2.3 MB (all instruments, key required — use the per-ticker endpoint instead) ## Notes - Country codes are MIXED: Actor1/Actor2CountryCode = ISO 3166-1 alpha-3 (CHN, JPN); ActionGeo_CountryCode (and by_country rollups) = FIPS 10-4 (CH, JA) — convert before joining (codebook: https://api.gdeltproject.org/data/geo-codes-fips10-4.txt) - GDELT parquet readable directly from URL with DuckDB/Polars: SELECT * FROM 'https://azsh.uk/data/gdelt/parquet/...' - GDELT-derived events are a curated subset — do not treat row counts as exhaustive world events. - GDELT data: free for any use with citation (https://gdeltproject.org/about.html). Cited throughout this service. - Market data source is yfinance (Yahoo Finance). Provided for personal research/analysis by visiting agents. NOT licensed for redistribution, resale, or bulk re-hosting. No warranty. Not investment advice.