Data integrity policy
KOUSOU 0 publishes 247,063,315 speech rows from 1,788 assemblies (1,741 municipalities and 47 prefectures), counted on 2026-08-18. We continuously verify where the data comes from, when it was retrieved, and whether the content is authentic, and disclose every correction as a full historical record.
Acquisition-time reliability
1. Source transparency
Each municipality's source URL and minutes system type (sophia / ssp / voices / dbsearch / kensakusystem / universal / family / voicetechno — 8 system families) is recorded in municipality_registry.jsonand publicly disclosed.
2. Original document preservation
Every source PDF / HTML is stored in Google Drive and Google Cloud Storage with a SHA-256 hash. Each speech row can be traced back to its original file via source_doc_id.
3. Unique municipality identifier
council_id is generated as a SHA-256 derivative to prevent collisions. To avoid mistaking same-named municipalities (e.g. Shimizu-cho in Hokkaido vs Shizuoka), source URLs are managed independently per municipality and a dictionary of 18 known traps is continuously maintained.
Automated integrity verification
1. Load-time verification
Immediately before each BigQuery write, the data is scored against whether it contains the canonical municipality name and prefecture name. Writes are aborted when the threshold is not met.
| Pattern | Score |
|---|---|
| Both municipality name + prefecture name | STRONG (1.0) |
| Prefecture name only | MEDIUM (0.5) |
| Municipality name only | WEAK (0.5) |
| Neither | NONE (0.0) |
Threshold: sample size × 0.3. Below this the write is aborted. Implementation: verify_council_name_match() inscripts/framework/council_name_verify.py.
2. Monthly full-fleet audit
At 04:00 JST on the first day of every month all 1,788 assemblies are mechanically audited (scripts/audit_bq_council_name_match.py). Any flagged inconsistency is investigated and corrected within 24 hours.
3. Additional evidence
- Council size vs municipality population sanity check
- Speech-date year-range sanity check
- speech_order continuity check (
continuous_quality_views.sql) - Duplicate detection (
dedupe_speeches.py)
Full disclosure of corrections
Correction logs reflect self-reported errors and corrective effort by KOUSOU 0; they do not indicate the quality or merit of any local council.
No corrections are currently published.
Third-party verifiability
- Each speech's
source_urlpoints directly to its public source - Each speech's
source_doc_idresolves to its archived original PDF (via thesource_documentstable) - Verification scripts are published on GitHub and can be re-run by anyone
- The BigQuery table
correlate-workspace.isvd_machikarte.speechesis read-accessible to approved AI agents and researchers
Quantitative quality indicators (as of 2026-06-02)
| Indicator | Value |
|---|---|
| Assemblies with speech data | 1,757 / 1,788 (98%) |
| Speech rows | 247,063,315 |
| Coverage period | 2010–2026 |
| Source systems | 8 families |
| Audit-passing (high-confidence) | 179 municipalities |
| Corrections (cumulative) | 0 cases / 0 rows (0.000%) |
| Monthly automated audit | Active since 2026-06-02 |
Continuous improvement
- Monthly audit results posted to Discord
- Same-name trap dictionary (
known_same_name_traps.json) continuously updated (currently 18 entries) - Audit algorithm refinement + manual spot checks to improve precision
- Inconsistency reports from users / researchers answered within 24 hours
Known limitations
The automated audit's scoring method has known room for improvement, and certain municipalities are difficult to evaluate purely mechanically due to the nature of their minutes. These are addressed through manual spot checks and algorithm refinement. Improvement status is updated continuously on this page.
Found an error? Please contact us via the corrections form. We respond within 24 hours.
Last updated: 2026-06-02
