SDN CORPUS TRANSPARENCY LOG
===========================

WHAT THIS IS
------------
A tamper-evident, append-only log of the raw OFAC "Specially Designated
Nationals" file (sdn_advanced.xml). Every 6 hours an observer fetches

    https://www.treasury.gov/ofac/downloads/sanctions/1.0/sdn_advanced.xml

and records the SHA-256 of the RAW BYTES, computed BEFORE any XML parsing.
One line is appended to corpus_chain.jsonl per observation. When the bytes
change, the full file is archived (xz-compressed) under archive/.

The point: from the log's first entry onward there is an accumulating,
hash-chained, externally-timestamped record of exactly which bytes were seen
and when. Anyone can re-verify it with no account and no wallet.

FILES HERE
----------
  corpus_chain.jsonl   the chain, one JSON object per line, oldest first
  archive/             xz-compressed copies of each DISTINCT raw file seen
                       (named <corpus_sha256>.xml.xz)
  heads/               <seq>.txt   = the entry's h (head hash)
                       <seq>.txt.ots = its OpenTimestamps proof
  verify_chain.py      the reference verifier (stdlib + rfc8785 + ots)
  README.txt           this file

CANONICALISATION
----------------
The per-entry hash h is computed over RFC 8785 (JCS) canonical JSON, NOT
Python's json.dumps. Every entry records  "canon":"RFC8785"  so this is
explicit and reproducible. h = sha256( RFC8785( entry without the "h" field ) ).

FIELDS (per line)
-----------------
  seq            sequence number, 1,2,3,... no gaps
  prev           previous entry's h; null for seq 1 (this is the chain link)
  observed_at    UTC ISO-8601, when the observation was recorded
  mode           "own-fetch" — the observer downloaded the URL itself
  verified_by    "full-hash" : the full body was downloaded this run and the
                               raw bytes were hashed directly.
                 "etag-304"  : the server returned HTTP 304 Not Modified for
                               the previous ETag, so NO bytes were downloaded.
                               corpus_sha256 and byte_length are CARRIED
                               FORWARD from the previous entry on the SERVER's
                               ETag assertion — this is NOT a fresh byte-level
                               re-hash. (seq 1-3 predate this field; they were
                               full 200 downloads.)
  source_url     the Treasury URL fetched
  http_status    200 (full download) or 304 (not modified)
  etag           the Treasury ETag header (opaque; NOT a content hash)
  last_modified  the Treasury Last-Modified header
  corpus_sha256  sha256 of the RAW xml bytes, before parsing
  byte_length    raw byte count
  changed        true if corpus_sha256 differs from the previous entry;
                 when true, the raw file was archived under
                 archive/<corpus_sha256>.xml.xz
  canon          always "RFC8785"
  h              sha256 over RFC8785(entry without "h")

TIMESTAMPING (OpenTimestamps)
-----------------------------
Each head hash h is written to heads/<seq>.txt and stamped with
OpenTimestamps (heads/<seq>.txt.ots), which anchors it into the Bitcoin
blockchain. That makes back-dating impossible: an entry's h provably existed
at or before the time its OTS proof confirms.

  * seq 1-3 PREDATE the OpenTimestamps integration and are UNSTAMPED
    (no heads/ files for them).
  * seq 4 is the ANCHORED GENESIS — the first head with an OTS proof.
  * A fresh stamp is "pending" (calendar commitment only) until its Bitcoin
    attestation confirms, typically a few hours; a weekly job upgrades the
    .ots files in place once confirmed.

HOW TO VERIFY (copy-paste; no account, no wallet)
-------------------------------------------------
  BASE=https://sanctions.nsgoods.org/sdn-log

  # 1) Re-hash the chain and check linkage (self-contained, no trust in us)
  curl -sO $BASE/corpus_chain.jsonl
  pip install rfc8785
  python3 - <<'EOF'
  import json, hashlib, rfc8785
  prev=None; exp=1
  for line in open("corpus_chain.jsonl"):
      line=line.strip()
      if not line: continue
      e=json.loads(line)
      assert e["seq"]==exp, ("seq gap at", e["seq"])
      assert e["prev"]==prev, ("prev linkage broken at", e["seq"])
      body={k:v for k,v in e.items() if k!="h"}
      h=hashlib.sha256(rfc8785.dumps(body)).hexdigest()
      assert h==e["h"], ("h mismatch at", e["seq"])
      prev=e["h"]; exp+=1
  print("CHAIN OK:", exp-1, "entries, head", prev)
  EOF

  # 2) Confirm an archived snapshot's bytes match the hash the log claims
  SHA=e71e2d07019500cdf1b39e74b31044aaa18499f06db60906cc8bc30765af548e
  curl -sO $BASE/archive/$SHA.xml.xz
  xz -dc $SHA.xml.xz | sha256sum
  # must print:  $SHA

  # 3) Verify an OpenTimestamps proof for a head (needs opentimestamps-client)
  pip install opentimestamps-client
  curl -sO $BASE/heads/4.txt
  curl -sO $BASE/heads/4.txt.ots
  ots verify 4.txt
  # and confirm heads/4.txt equals the "h" of seq 4 in corpus_chain.jsonl

  # (Optional) run our reference verifier over a full local copy:
  #   download corpus_chain.jsonl, verify_chain.py, heads/*, archive/* into one
  #   directory (keeping the heads/ and archive/ subdirs), then:
  #   pip install rfc8785 opentimestamps-client && python3 verify_chain.py

WHAT THIS PROVES — AND WHAT IT DOES NOT
---------------------------------------
PROVES: the log has not been rewritten after the fact. The hash chain plus the
OpenTimestamps proofs anchor each head to a time it demonstrably existed, so
entries cannot be silently altered or back-dated later.

DOES NOT PROVE: that an archived file is Treasury's genuine sdn_advanced.xml.
Treasury does not sign the file, and this log is produced by a single observer.
We assert we fetched the bytes from the Treasury URL, but a determined operator
could have recorded a substituted file. To gain independent confidence, compare
a corpus_sha256 here against your OWN fetch of the Treasury URL at the same
time, or against another independent observer. Also note "etag-304" entries
assert "unchanged" on the server's ETag, not on a fresh byte re-hash.