Publish the reproducible Sound Lab and FPGA Colour Lab sources

This commit is contained in:
bholman committed 2026-11-18 12:28:31 -07:00
commit 88254d7e24
27 files changed
+4574

No files matched your search

+157
View File
@@ -0,0 +1,157 @@
# Sound Lab host reports
`python3 soundlab_reports.py` requires Python 3.10 or later and only the standard
library. It reads existing disk images and report files; it does not run the
emulator, touch a mounted disk, contact a service or modify source inputs.
## Extract and preserve reports
Quit or safely eject/save the writable diagnostic disk first, then copy it to a
host working folder. Extract from that saved copy:
```sh
python3 examples/sound-lab/host/soundlab_reports.py extract /path/to/SOUNDLAB.DSK \
--output /path/to/run-exports
```
The input must be a raw, single-sided, standard 35-track Disk BASIC disk image:
35 tracks × 18 sectors × 256 bytes = 161,280 bytes. DMK, images with headers and
other geometries are rejected rather than guessed. Selected filenames are
`SND0000.TXT` through `SND9999.TXT`, case-insensitively. Reports must have the
Disk BASIC ASCII flag `FF`; the guest writes directory type 3 (text). The tool
preserves every selected file's exact bytes, normalizes its output filename to
uppercase, and writes `extraction.json` with the original disk's SHA-256, each
report's SHA-256/byte count and its granule chain.
Validation covers the directory, file type/ASCII flag, final-sector byte count,
FAT values, fragmented chains, cycles, free granules in a file chain, duplicate
names, unsafe names and crosslinks involving *any* active file. The directory
track is excluded from granule addressing. Unclaimed FAT allocations are allowed
because interrupted writes can leave them; the extractor does not repair them.
It validates filesystem structure, not the authenticity of the disk's contents.
An existing output is never replaced implicitly. Prefer a fresh output folder
for each run. `--overwrite` permits replacement of ordinary output files only;
symlinks, inputs and hardlinks to inputs are still rejected. Keep the source
`.DSK` and extracted `.TXT` as original evidence. The extractor never reformats,
repairs or writes back to the disk.
## Convert and compare
```sh
python3 examples/sound-lab/host/soundlab_reports.py convert \
/path/to/run-exports/SND0001.TXT --output /path/to/readable-run \
--manifest .build/sound-lab/SOUNDLAB-manifest.json
python3 examples/sound-lab/host/soundlab_reports.py compare \
/path/to/vcc/SND0001.TXT /path/to/real-coco/SND0001.TXT \
--output /path/to/comparison \
--manifest .build/sound-lab/SOUNDLAB-manifest.json
```
Conversion writes JSON, CSV and a self-contained readable HTML file. The JSON
retains all recorded run metadata and per-case fields; CSV keeps raw timing/status
fields as columns. The HTML escapes report text and includes no scripts or remote
assets. No audio files or listening claims are manufactured.
The optional authoritative manifest has this supported structure (other fields
such as artifact hashes may also be present):
```json
{
"build_id": "BUILD-IDENTITY",
"vector_set_crc16": "1234",
"vectors": [{"case_id": "TON01-A", "crc16": "7ABC"}]
}
```
The vector list may alternatively be an object such as
`"vectors": {"TON01-A": "7ABC"}`. CRC values in the manifest may be four-digit
hex strings, strings prefixed with `0x`, or JSON integers. Use ordinary four-digit
hex in report records. Uppercase manifest keys `BUILD_ID` and
`VECTOR_SET_CRC16` are accepted aliases. A manifest binds the guest's short build
and vector IDs to the host build receipt; a CRC is not a cryptographic signature.
The separate artifact hashes establish which binary/disk bytes were built.
Comparison requires the same nonempty `BUILD_ID`, `VECTOR_SET_CRC16` and matching
per-case `VECTOR_CRC16`. With a manifest, both reports must match its declared
identities too. Without one, the result explicitly states
`identity_basis=REPORT_CLAIMS_ONLY`. Different/missing identities, missing cases,
partial reports or failed integrity checks produce `NOT_COMPARABLE`, with the
recorded evidence still shown. Comparable cases show `SAME` or `DIFFERENT` and
list differing metrics, raw status, timings or notes. Matching these values does
**not** establish matching audible output or equal analog timing precision.
`AUTO` and `HUMAN` are separate columns. `AUTO=OBSERVED` never becomes
`HUMAN=HEARD`. Missing outcomes remain `UNKNOWN`; unknown timing resolution stays
unknown. The tool checks report consistency and identities; it does not certify
the author's automatic verdict or make hardware pass/fail assertions.
Exit status is 0 for successful valid conversion/comparison or extraction, 1 for
an unreadable/unsupported input or refused output, and 2 when result files were
written but evidence is partial, invalid or not comparable. Inspect the JSON/HTML
issues instead of treating exit 2 as a completed run. Optional absent checksum or
count fields are explicitly `NOT_PROVIDED`, never `VERIFIED`.
## SOUNDLAB-1 record contract
The first nonblank line is exactly `FORMAT=SOUNDLAB-1`. Subsequent ordinary lines
contain space-separated `KEY=VALUE` tokens with uppercase keys and nonempty values.
Values containing whitespace belong in escaped `NOTE` records. CRLF is preferred;
LF is accepted. Bare CR, non-ASCII bytes and unescaped control characters are
rejected. Empty/whitespace-only lines are ignored for record counts and checksums.
Unknown metadata and per-case keys are retained for forward-compatible evidence.
A different `FORMAT` requires a new parser version.
A header declares the run, target and build/vector identity. A case starts with
`CASE=... VECTOR_CRC16=HHHH AUTO=... HUMAN=...`; subsequent lines carry its metrics.
`ENDCASE` must match that case's identity. A per-case `REPEAT=0` field identifies
the first attempt, `REPEAT=1` the next, and so on. Each `(CASE, REPEAT)` pair must
be unique within a report. Omitted `REPEAT` defaults to 0 for early SOUNDLAB-1
compatibility. Attempts are retained separately in JSON/CSV/HTML and compared by
that pair, even if recorded in a different order. An interrupted case/run is preserved
and marked incomplete. Case IDs and outcome enumerations compare
case-insensitively; the original text/fields remain available.
Allowed automatic values: `PASS`, `FAIL`, `OBSERVED`, `TIMEOUT`, `NOT_RUN`,
`NOT_APPLICABLE`. Allowed human values: `HEARD`, `WRONG`, `NO_SOUND`, `UNSURE`,
`NOT_OBSERVED`, `SKIPPED`.
An optional note consists of two consecutive records inside a case:
```text
NOTE_LENGTH=18
NOTE=FIRST LINE\nSECOND!\n
```
`NOTE_LENGTH` counts decoded ASCII **bytes**, not the serialized escape length.
Escapes are `\\`, `\n`, `\r`, `\t` and `\xHH` (ASCII 00–7F). `NOTE=` consumes the
rest of its line, including spaces. Length mismatch, unknown escapes, a dangling
length or a note outside a case is reported as invalid/incomplete evidence.
A finished run ends with one footer, with no following nonblank records:
```text
ENDRUN=0001 COMPLETE=YES RECORD_COUNT=8 CHECKSUM_CRC16=1234
```
The run number must match `RUN`. `COMPLETE=NO` explicitly retains an interrupted
run. Footer integrity fields are optional for interoperability; the diagnostic
writer supplies both. `RECORD_COUNT` is the number of preceding nonblank lines,
including `FORMAT` and every case, note and `ENDCASE` line, excluding the footer.
`CHECKSUM_CRC16` is CRC-16/CCITT-FALSE: polynomial `1021`, initial `FFFF`, no
reflection, no final XOR. It covers those same original nonblank lines joined
with CRLF, **including a final CRLF**, and excludes the footer. Lowercase and
uppercase hex are equivalent. Example footer values above are schematic.
## ROM-free checks
```sh
python3 -m unittest discover -s examples/sound-lab/tests -p test_reports.py -v
```
The authored fixtures cover fragmented disks across the reserved track, full
sectors/granules, truncation, malformed FAT and loops/crosslinks, report end
markers, notes, count/checksum validation, independent outcome comparisons,
manifest identities, raw metric differences and output preservation. They use no
private ROMs, real reports or actual hardware recordings.
+525
View File
@@ -0,0 +1,525 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-3.0-or-later
"""Read-only Sound Lab Disk BASIC extraction and evidence reporting (stdlib only)."""
from __future__ import annotations
import argparse
import csv
import hashlib
import html
import io
import json
from pathlib import Path
import re
import sys
from typing import Any
DISK_SIZE = 35 * 18 * 256
FAT_OFFSET = (17 * 18 + 1) * 256
DIRECTORY_OFFSET = (17 * 18 + 2) * 256
REPORT_NAME = re.compile(r"SND[0-9]{4}\.TXT", re.IGNORECASE)
KEY = re.compile(r"[A-Z][A-Z0-9_]*")
AUTO = frozenset("PASS FAIL OBSERVED TIMEOUT NOT_RUN NOT_APPLICABLE".split())
HUMAN = frozenset("HEARD WRONG NO_SOUND UNSURE NOT_OBSERVED SKIPPED".split())
class SoundLabError(ValueError):
"""An invalid disk/report or unsafe output operation."""
def crc16(data: bytes) -> int:
"""CRC-16/CCITT-FALSE: polynomial 1021, initial FFFF, no reflection/xor."""
result = 0xFFFF
for value in data:
result ^= value << 8
for _ in range(8):
result = ((result << 1) ^ (0x1021 if result & 0x8000 else 0)) & 0xFFFF
return result
def granule_offset(granule: int) -> int:
track = granule // 2 + (1 if granule >= 34 else 0)
return (track * 18 + (granule % 2) * 9) * 256
def extract_reports(disk: bytes) -> list[dict[str, Any]]:
"""Validate active DECB files, return SNDnnnn.TXT ASCII bytes; never mutate disk.
All active files are walked to catch crosslinks into a selected report. Unused
FAT allocations are tolerated: they can remain after an interrupted write.
"""
if len(disk) != DISK_SIZE:
raise SoundLabError(f"Expected raw 35-track DSK ({DISK_SIZE} bytes), got {len(disk)}")
fat = disk[FAT_OFFSET:FAT_OFFSET + 68]
for index, value in enumerate(fat):
if not (value < 68 or 0xC0 <= value <= 0xC9 or value == 0xFF):
raise SoundLabError(f"Invalid FAT value {value:02X} at granule {index}")
owned: dict[int, str] = {}
names: set[str] = set()
reports = []
for index in range(72):
entry = disk[DIRECTORY_OFFSET + index * 32:DIRECTORY_OFFSET + (index + 1) * 32]
if entry[0] in (0, 0xFF):
continue
try:
base = entry[:8].decode("ascii").rstrip(" ")
ext = entry[8:11].decode("ascii").rstrip(" ")
except UnicodeDecodeError as error:
raise SoundLabError(f"Non-ASCII directory name at entry {index}") from error
if not base or any(ord(c) < 33 or c in '/\\:.' for c in base + ext):
raise SoundLabError(f"Unsafe directory name at entry {index}")
name = f"{base}.{ext}" if ext else base
normalized = name.upper()
if normalized in names:
raise SoundLabError(f"Duplicate directory name: {name}")
names.add(normalized)
if entry[11] > 3 or entry[12] not in (0, 0xFF):
raise SoundLabError(f"Invalid file type/ASCII flag for {name}")
last_bytes = int.from_bytes(entry[14:16], "big")
if last_bytes > 256:
raise SoundLabError(f"Invalid final-sector byte count for {name}")
chain = []
current = entry[13]
while True:
if current >= 68:
raise SoundLabError(f"Out-of-range granule {current} for {name}")
if current in chain:
raise SoundLabError(f"FAT loop for {name}")
if current in owned:
raise SoundLabError(f"Crosslinked granule {current}: {name} and {owned[current]}")
chain.append(current)
owned[current] = name
value = fat[current]
if 0xC0 <= value <= 0xC9:
final_sectors = value - 0xC0
if final_sectors == 0 and (len(chain) != 1 or last_bytes != 0):
raise SoundLabError(f"Invalid empty final granule for {name}")
if final_sectors > 0 and last_bytes == 0:
# DOS stores 256, not zero, for a completely used last sector.
raise SoundLabError(f"Missing final-sector bytes for {name}")
break
if value == 0xFF:
raise SoundLabError(f"Allocated file {name} points to free granule {current}")
current = value
length = (len(chain) - 1) * 2304
if final_sectors:
length += (final_sectors - 1) * 256 + last_bytes
if REPORT_NAME.fullmatch(name):
if entry[12] != 0xFF:
raise SoundLabError(f"Report {name} is not marked Disk BASIC ASCII")
content = b"".join(disk[granule_offset(g):granule_offset(g) + 2304] for g in chain)[:length]
reports.append({"filename": normalized, "data": content,
"byte_count": len(content), "sha256": hashlib.sha256(content).hexdigest(),
"granules": chain, "directory_type": entry[11]})
return sorted(reports, key=lambda item: item["filename"])
def decode_note(value: str) -> str:
decoded = bytearray()
i = 0
while i < len(value):
char = value[i]
if char != "\\":
if not 32 <= ord(char) <= 126:
raise SoundLabError("Notes must use printable ASCII or escaped control bytes")
decoded.append(ord(char))
i += 1
continue
i += 1
if i >= len(value):
raise SoundLabError("Incomplete note escape")
escaped = value[i]
if escaped in "\\nrt":
decoded.append({"\\": 92, "n": 10, "r": 13, "t": 9}[escaped])
i += 1
elif escaped == "x" and i + 2 < len(value) and re.fullmatch(r"[0-9A-Fa-f]{2}", value[i + 1:i + 3]):
byte = int(value[i + 1:i + 3], 16)
if byte > 127:
raise SoundLabError("Escaped notes must remain ASCII")
decoded.append(byte)
i += 3
else:
raise SoundLabError(f"Unknown note escape at position {i}")
return decoded.decode("ascii")
def parse_tokens(line: str) -> dict[str, str]:
fields = {}
for token in line.split():
if "=" not in token:
raise SoundLabError(f"Expected KEY=VALUE token: {token}")
key, value = token.split("=", 1)
if not KEY.fullmatch(key) or not value or key in fields:
raise SoundLabError(f"Invalid or duplicate token: {token}")
fields[key] = value
if not fields:
raise SoundLabError("Empty record")
return fields
def parse_report(data: bytes, source: str = "") -> dict[str, Any]:
"""Parse evidence without turning automatic observations into listening results.
Incomplete files are useful evidence, returned with complete=False and issues.
Structural and integrity failures likewise remain visible in JSON/HTML; a
malformed header/encoding is rejected because it is not a SOUNDLAB-1 report.
"""
try:
text = data.decode("ascii")
except UnicodeDecodeError as error:
raise SoundLabError("Report is not ASCII") from error
if "\x00" in text or re.search(r"[\x01-\x08\x0b\x0c\x0e-\x1f\x7f]", text):
raise SoundLabError("Report contains unescaped control bytes")
if "\r" in text.replace("\r\n", ""):
raise SoundLabError("Report has bare CR line endings")
lines = [line for line in text.replace("\r\n", "\n").split("\n") if line.strip()]
if not lines or lines[0] != "FORMAT=SOUNDLAB-1":
raise SoundLabError("Missing supported FORMAT=SOUNDLAB-1 header")
result: dict[str, Any] = {"format": "SOUNDLAB-1", "source": source,
"source_sha256": hashlib.sha256(data).hexdigest(),
"metadata": {}, "cases": [], "issues": [], "complete": False,
"integrity": {"record_count": "NOT_PROVIDED", "checksum_crc16": "NOT_PROVIDED"}}
current = None
footer = None
note_length = None
records = []
def issue(message: str) -> None:
result["issues"].append(message)
def merge(target: dict[str, str], fields: dict[str, str], number: int) -> None:
for key, value in fields.items():
if key in target:
issue(f"Line {number}: duplicate field {key}")
else:
target[key] = value
for number, line in enumerate(lines, 1):
if footer is not None:
issue(f"Line {number}: record after ENDRUN")
continue
if line.startswith("NOTE="):
records.append(line)
try:
note = decode_note(line[5:])
if current is None:
issue(f"Line {number}: NOTE outside CASE")
elif note_length is None:
issue(f"Line {number}: NOTE without NOTE_LENGTH")
else:
current["notes"].append(note)
if len(note.encode("ascii")) != note_length:
issue(f"Line {number}: NOTE_LENGTH mismatch")
except SoundLabError as error:
issue(f"Line {number}: {error}")
note_length = None
continue
if note_length is not None:
issue(f"Line {number}: NOTE_LENGTH not followed by NOTE")
note_length = None
try:
fields = parse_tokens(line)
except SoundLabError as error:
records.append(line)
issue(f"Line {number}: {error}")
continue
controls = set(fields) & {"FORMAT", "RUN", "CASE", "ENDCASE", "ENDRUN", "NOTE_LENGTH"}
if len(controls) > 1:
issue(f"Line {number}: conflicting control records")
if "FORMAT" in fields and number != 1:
issue(f"Line {number}: repeated FORMAT header")
if "RUN" in fields and (current is not None or result["cases"]):
issue(f"Line {number}: RUN header after CASE")
if "ENDRUN" in fields:
footer = fields
result["footer"] = fields
if current is not None:
issue(f"Line {number}: ENDRUN before ENDCASE")
if fields["ENDRUN"] != result["metadata"].get("RUN"):
issue(f"Line {number}: ENDRUN does not match RUN")
if fields.get("COMPLETE", "").upper() not in ("YES", "NO"):
issue(f"Line {number}: COMPLETE must be YES or NO")
continue
records.append(line)
if number == 1:
continue
if "CASE" in fields:
if current is not None:
issue(f"Line {number}: CASE before previous ENDCASE")
identifier = fields["CASE"].upper()
current = {"id": identifier, "fields": {}, "notes": [], "closed": False}
result["cases"].append(current)
merge(current["fields"], fields, number)
elif "ENDCASE" in fields:
if current is None:
issue(f"Line {number}: ENDCASE outside CASE")
else:
if fields["ENDCASE"].upper() != current["id"]:
issue(f"Line {number}: ENDCASE does not match CASE")
else:
current["closed"] = True
current = None
if len(fields) != 1:
issue(f"Line {number}: unexpected ENDCASE fields")
elif "NOTE_LENGTH" in fields:
if len(fields) != 1 or current is None:
issue(f"Line {number}: NOTE_LENGTH must be alone inside CASE")
try:
note_length = int(fields["NOTE_LENGTH"])
if note_length < 0 or note_length > 65535:
raise ValueError()
except ValueError:
note_length = None
issue(f"Line {number}: invalid NOTE_LENGTH")
else:
merge(current["fields"] if current is not None else result["metadata"], fields, number)
if note_length is not None:
issue("Truncated NOTE after NOTE_LENGTH")
if not result["metadata"].get("RUN"):
issue("Missing RUN identity")
attempts: set[tuple[str, int]] = set()
for case in result["cases"]:
try:
repeat_text = case["fields"].get("REPEAT", "0")
if not re.fullmatch(r"[0-9]+", repeat_text):
raise ValueError()
repeat = int(repeat_text)
if repeat > 65535:
raise ValueError()
case["repeat"] = repeat
identity = (case["id"], repeat)
if identity in attempts:
issue(f"CASE {case['id']}: duplicate REPEAT={repeat}")
attempts.add(identity)
except ValueError:
case["repeat"] = None
issue(f"CASE {case['id']}: invalid REPEAT")
for key, allowed in (("AUTO", AUTO), ("HUMAN", HUMAN)):
value = case["fields"].get(key)
if value is None:
# UNKNOWN is deliberate: absence is not a positive or negative observation.
case[key.lower()] = "UNKNOWN"
issue(f"CASE {case['id']}: missing {key}")
else:
case[key.lower()] = value.upper()
if value.upper() not in allowed:
issue(f"CASE {case['id']}: unknown {key} value {value}")
vector_crc = case["fields"].get("VECTOR_CRC16", "")
if not re.fullmatch(r"[0-9A-Fa-f]{4}", vector_crc):
issue(f"CASE {case['id']}: missing or invalid VECTOR_CRC16")
if not case["closed"]:
issue(f"CASE {case['id']}: missing ENDCASE")
if footer is None:
issue("Missing ENDRUN; partial run")
else:
if "RECORD_COUNT" in footer:
try:
valid_count = int(footer["RECORD_COUNT"]) == len(records)
except ValueError:
valid_count = False
result["integrity"]["record_count"] = "VERIFIED" if valid_count else "FAILED"
if not valid_count:
issue("RECORD_COUNT mismatch")
if "CHECKSUM_CRC16" in footer:
canonical = ("\r\n".join(records) + "\r\n").encode("ascii")
expected = f"{crc16(canonical):04X}"
valid_crc = footer["CHECKSUM_CRC16"].upper() == expected
result["integrity"]["checksum_crc16"] = "VERIFIED" if valid_crc else "FAILED"
if not valid_crc:
issue(f"CHECKSUM_CRC16 mismatch (calculated {expected})")
if footer.get("COMPLETE", "").upper() != "YES":
issue("Run marked incomplete")
result["record_count"] = len(records)
result["complete"] = footer is not None and footer.get("COMPLETE", "").upper() == "YES" and not result["issues"]
result["valid"] = not result["issues"]
return result
def validate_manifest(report: dict[str, Any], manifest: dict[str, Any]) -> list[str]:
"""Return exact identity failures; a manifest is evidence, not a trust signature."""
if not isinstance(manifest, dict):
raise SoundLabError("Manifest must be a JSON object")
problems = []
metadata = report["metadata"]
def crc_text(value: Any) -> str:
if isinstance(value, int):
return f"{value:04X}"
return str(value).upper().removeprefix("0X")
for key in ("BUILD_ID", "VECTOR_SET_CRC16"):
expected = manifest.get(key.lower(), manifest.get(key))
expected_text = crc_text(expected) if key.endswith("CRC16") else str(expected).upper()
if expected is None or expected_text != metadata.get(key, "").upper():
problems.append(f"{key} differs from manifest or is absent")
vectors = manifest.get("vectors", {})
if isinstance(vectors, list):
if not all(isinstance(item, dict) for item in vectors):
raise SoundLabError("Manifest vectors must be objects")
vectors = {item.get("id", item.get("case_id", "")): item.get("crc16", item.get("vector_crc16", "")) for item in vectors}
if not isinstance(vectors, dict):
raise SoundLabError("Manifest vectors must be a list or object")
vectors = {str(key).upper(): crc_text(value) for key, value in vectors.items()}
for case in report["cases"]:
expected = vectors.get(case["id"])
if expected is None or expected != case["fields"].get("VECTOR_CRC16", "").upper():
problems.append(f"CASE {case['id']}: vector differs from manifest or is absent")
return problems
def compare_reports(left: dict[str, Any], right: dict[str, Any], manifest: dict[str, Any] | None = None) -> dict[str, Any]:
problems = []
for key in ("BUILD_ID", "VECTOR_SET_CRC16"):
a, b = left["metadata"].get(key), right["metadata"].get(key)
if not a or not b or a.upper() != b.upper():
problems.append(f"{key} differs or is absent")
if manifest is not None:
problems.extend("Left: " + value for value in validate_manifest(left, manifest))
problems.extend("Right: " + value for value in validate_manifest(right, manifest))
if not left["valid"] or not right["valid"]:
problems.append("One or both reports are partial or invalid; outcomes are provisional")
by_id = [{(case["id"], case["repeat"]): case for case in report["cases"]} for report in (left, right)]
rows = []
for identity in sorted(set(by_id[0]) | set(by_id[1]), key=lambda item: (item[0], -1 if item[1] is None else item[1])):
identifier, repeat = identity
a, b = by_id[0].get(identity), by_id[1].get(identity)
reason = list(problems)
if a is None or b is None:
reason.append("CASE missing from one report")
elif a["fields"].get("VECTOR_CRC16", "").upper() != b["fields"].get("VECTOR_CRC16", "").upper():
reason.append("VECTOR_CRC16 differs")
row = {"case": identifier, "repeat": repeat, "left_auto": a["auto"] if a else "UNKNOWN",
"right_auto": b["auto"] if b else "UNKNOWN", "left_human": a["human"] if a else "UNKNOWN",
"right_human": b["human"] if b else "UNKNOWN", "reasons": reason}
changes = {}
if a is not None and b is not None:
for key in sorted(set(a["fields"]) | set(b["fields"])):
if key in ("CASE", "REPEAT", "VECTOR_CRC16", "AUTO", "HUMAN"):
continue
av, bv = a["fields"].get(key), b["fields"].get(key)
if av != bv:
changes[key] = {"left": av, "right": bv}
if a["notes"] != b["notes"]:
changes["NOTES"] = {"left": a["notes"], "right": b["notes"]}
row["recorded_differences"] = changes
outcomes_equal = row["left_auto"] == row["right_auto"] and row["left_human"] == row["right_human"]
row["comparison"] = "NOT_COMPARABLE" if reason else ("SAME" if outcomes_equal and not changes else "DIFFERENT")
rows.append(row)
return {"left": left["source"], "right": right["source"], "identity_issues": problems,
"manifest_checked": manifest is not None, "identity_basis": "MANIFEST" if manifest is not None else "REPORT_CLAIMS_ONLY", "cases": rows,
"notice": "Matching automatic observations do not establish audible equivalence. HUMAN is independent."}
def render_html(document: dict[str, Any]) -> str:
esc = lambda value: html.escape(str(value))
comparison = "left" in document
title = "Sound Lab comparison" if comparison else "Sound Lab report"
pieces = [f'<!doctype html><html lang="en"><meta charset="utf-8"><meta name="viewport" content="width=device-width"><title>{title}</title>',
'<style>body{font:16px system-ui;max-width:75rem;margin:2rem auto;padding:0 1rem;line-height:1.5;color:#20252b;background:#fff}table{border-collapse:collapse;width:100%}td,th{padding:.5rem;border:1px solid #999;text-align:left;vertical-align:top}pre{white-space:pre-wrap;overflow-wrap:anywhere}.warning{background:#fff0c2;padding:1rem}code{overflow-wrap:anywhere}</style>', f'<h1>{title}</h1>', '<p>Automatic observations and human listening results are separate evidence. Unknown means unreported; it never means heard.</p>']
issues = document.get("identity_issues", document.get("issues", []))
if issues:
pieces.append('<div class="warning"><strong>Incomplete or unverified evidence</strong><ul>' + ''.join(f'<li>{esc(value)}</li>' for value in issues) + '</ul></div>')
if comparison:
keys = ["case", "repeat", "left_auto", "right_auto", "left_human", "right_human", "comparison", "recorded_differences", "reasons"]
rows = document["cases"]
else:
pieces.append(f'<p>Source: <code>{esc(document["source"])}</code><br>SHA-256: <code>{esc(document["source_sha256"])}</code><br>Complete: {esc(document["complete"])}</p>')
pieces.append('<h2>Run metadata and integrity</h2><pre>' + esc(json.dumps({"metadata": document["metadata"], "integrity": document["integrity"]}, indent=2)) + '</pre>')
keys = ["case", "repeat", "auto", "human", "closed", "notes", "recorded_fields"]
rows = [{"case": case["id"], "repeat": case["repeat"], "auto": case["auto"], "human": case["human"], "closed": case["closed"], "notes": "\n".join(case["notes"]), "recorded_fields": json.dumps(case["fields"], sort_keys=True)} for case in document["cases"]]
pieces.append('<h2>Cases</h2><table><thead><tr>' + ''.join(f'<th scope="col">{esc(key)}</th>' for key in keys) + '</tr></thead><tbody>')
for row in rows:
pieces.append('<tr>' + ''.join(f'<td><pre>{esc(row.get(key, ""))}</pre></td>' for key in keys) + '</tr>')
pieces.append('</tbody></table></html>')
return '\n'.join(pieces)
def render_csv(document: dict[str, Any]) -> str:
stream = io.StringIO(newline="")
if "left" in document:
fields = ["case", "repeat", "left_auto", "right_auto", "left_human", "right_human", "comparison", "recorded_differences", "reasons"]
rows = document["cases"]
else:
# Union preserves timing/status fields, rather than reducing reports to outcomes.
extra = sorted({key for case in document["cases"] for key in case["fields"]} - {"CASE", "REPEAT", "AUTO", "HUMAN"})
fields = ["case", "repeat", "auto", "human", "closed", "notes"] + extra
rows = [{"case": case["id"], "repeat": case["repeat"], "auto": case["auto"], "human": case["human"], "closed": case["closed"], "notes": '\n'.join(case["notes"]), **{key: case["fields"].get(key, "") for key in extra}} for case in document["cases"]]
writer = csv.DictWriter(stream, fieldnames=fields)
writer.writeheader()
writer.writerows(rows)
return stream.getvalue()
def write_outputs(outputs: dict[Path, bytes], overwrite: bool = False) -> None:
"""Preflight every target before any writes; originals are never implicitly replaced."""
for path in outputs:
if path.is_symlink() or (path.exists() and (not overwrite or not path.is_file())):
raise SoundLabError(f"Refusing to replace {path}; select a fresh folder or use --overwrite for regular files")
for path, data in outputs.items():
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("wb" if overwrite else "xb") as handle:
handle.write(data)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
sub = parser.add_subparsers(dest="operation", required=True)
for operation in ("extract", "convert", "compare"):
command = sub.add_parser(operation)
command.add_argument("source", type=Path)
if operation == "compare":
command.add_argument("other", type=Path)
command.add_argument("--output", required=True, type=Path)
command.add_argument("--overwrite", action="store_true")
if operation != "extract":
command.add_argument("--manifest", type=Path, help="Authoritative build/vector JSON manifest")
args = parser.parse_args(argv)
try:
outputs = {}
if args.operation == "extract":
disk = args.source.read_bytes()
reports = extract_reports(disk)
if not reports:
raise SoundLabError("No SNDnnnn.TXT ASCII reports found")
receipt = {"disk": args.source.name, "disk_byte_count": len(disk), "disk_sha256": hashlib.sha256(disk).hexdigest(), "reports": []}
for report in reports:
outputs[args.output / report["filename"]] = report["data"]
receipt["reports"].append({key: value for key, value in report.items() if key != "data"})
outputs[args.output / "extraction.json"] = (json.dumps(receipt, indent=2) + '\n').encode()
else:
left = parse_report(args.source.read_bytes(), args.source.name)
manifest = json.loads(args.manifest.read_text()) if args.manifest else None
if args.manifest and not isinstance(manifest, dict):
raise SoundLabError("Manifest must be a JSON object")
if args.operation == "compare":
document = compare_reports(left, parse_report(args.other.read_bytes(), args.other.name), manifest)
stem = "comparison"
else:
document = left
if manifest is not None:
document["manifest_issues"] = validate_manifest(document, manifest)
document["issues"].extend(document["manifest_issues"])
document["valid"] = document["valid"] and not document["manifest_issues"]
document["complete"] = document["complete"] and document["valid"]
stem = args.source.stem
outputs = {args.output / f"{stem}.json": (json.dumps(document, indent=2) + '\n').encode(), args.output / f"{stem}.csv": render_csv(document).encode(), args.output / f"{stem}.html": render_html(document).encode()}
# Never allow an output to alias any supplied source, even with --overwrite.
sources = [args.source] + ([args.other] if args.operation == "compare" else [])
if getattr(args, "manifest", None):
sources.append(args.manifest)
source_paths = {path.resolve() for path in sources}
if any(path.resolve() in source_paths or any(path.exists() and path.samefile(source) for source in sources) for path in outputs):
raise SoundLabError("Output would replace an input; choose another folder")
write_outputs(outputs, args.overwrite)
print(f"Wrote {len(outputs)} files to {args.output}")
if args.operation != "extract" and (document.get("issues") or document.get("identity_issues") or any(row.get("comparison") == "NOT_COMPARABLE" for row in document["cases"])):
print("Warning: evidence is incomplete, invalid, or not comparable; inspect JSON/HTML", file=sys.stderr)
return 2
return 0
except (OSError, SoundLabError, json.JSONDecodeError) as error:
print(f"soundlab: {error}", file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())