V2 Schema Changes
67 records
| id ▼ | ts | kind | before | after | rationale | n_docs_touched |
|---|---|---|---|---|---|---|
| 1 | 1778850823 | new_type | divider | Cover/divider/label pages with case identifier in handwriting on otherwise blank page. Currently labeled blank.white, image.scan, or misc. Seeds: EFTA00006369. | 0 | |
| 2 | 1778850823 | new_type | placeholder.native | DOJ "Native Placeholder — No Images Produced" pages. Structural artifact of mixed-format productions. Currently spread across misc, blank, and other catch-alls. Seeds: EFTA00276490. | 0 | |
| 3 | 1778850823 | new_type | court.reporter_log | Court reporter witness logs (typed form, hand-filled cells). Distinct from court.filing. Seeds: EFTA01245715. | 0 | |
| 4 | 1778850823 | new_flag | has_handwriting | Boolean — doc contains any meaningful handwritten content (annotation OR primary). Decoupled from doc_type so a typed-with-markup doc keeps its true type. | 0 | |
| 5 | 1778850823 | new_flag | has_letterhead | Doc is on identifiable letterhead (esp. "Jeffrey E. Epstein" stationery, seen in 00038379/00087956/01325170/02729170). | 0 | |
| 6 | 1778850823 | new_flag | is_personal_correspondence | Doc is a personal letter or note, not formal/business correspondence. | 0 | |
| 7 | 1778850823 | new_flag | is_investigator_notes | Handwritten notes by law-enforcement/investigator from a witness interview. Seeds: MCC investigation series 135084-135584. | 0 | |
| 8 | 1778850823 | new_flag | is_protected_material | Page bears "PROTECTED MATERIAL" or similar court-imposed marking. Worth surfacing. | 0 | |
| 9 | 1778850823 | new_flag | is_prosecution_prep | Prosecutor/AUSA case-prep notes, typically Jencks-marked. Seeds: EFTA01245499 (Tablet 01). | 0 | |
| 10 | 1778850823 | new_flag | is_investigator_annotated | Typed/printed doc with handwritten investigator markup overlaid (red pen, marginalia). Seeds: EFTA00263797. | 0 | |
| 11 | 1778850823 | split | handwritten | handwritten + flag(s) | The current `handwritten` doc_type was conflating: pure handwritten (interview notes, personal letters, Epstein contact lists), cover/divider pages (which are functionally `blank` or new `divider`), and typed docs with handwritten annotations (which keep their typed primary type + has_handwriting flag). v2: handwritten = primary content is handwritten; everything else demoted to flag. | 0 |
| 12 | 1778850823 | observation | image.scan (31.7K docs) | (heterogeneous: photographs / handwritten scans / scanned typed text) | image.scan is the dominant misclassification target for all 3 visual subgroups. 15/17 user-handwritten corrections came from image.scan. v2 priority: decompose image.scan into image.photo, handwritten, image.scanned_typed (with OCR-poor typed content). | 0 |
| 13 | 1778851055 | observation | BGE text-embedding NN for handwritten | ~50% precision on high-signal (2+ embeddings) | Text embeddings can't reliably distinguish handwritten from photo/blank because OCR text is uniformly noisy for all three. For better handwritten discovery, use: (1) CLIP visual embeddings (611K already indexed via /find-image), (2) Bates-range search within known investigation series (MCC notes form a contiguous range 135084-135584), (3) direct vision classification of v1=image.scan candidates. | 0 |
| 14 | 1778851425 | bulk_reassign | mixed (financial.equity_swap, memo, court.exhibit_list, ...) | fbi.evidence_receipt + is_fd1087 flag | 63 unique docs in corpus contain 'FD-1087' (FBI Collected Item Log). 37 already correctly labeled fbi.evidence_receipt; 24 were misclassified across financial/memo/court types. Reassigned all 24 via heuristic corpus text search. Seed: EFTA00027970 (user correction). | 24 |
| 15 | 1778851603 | schema_principle | rolled-up fbi.evidence_receipt | fbi.<form_number> — fbi.fd302, fbi.fd302a, fbi.fd1087, fbi.fd597, fbi.fd1023, fbi.fd340, fbi.fd71 etc. | Per user insight: the FBI categorizes its own paperwork by form number; v2 should mirror that distribution. v1 has form-based for some (fbi.302, fbi.302a, fbi.fincen) but rolls others (fbi.evidence_receipt catches FD-1087, FD-597, etc.). v2: carry the FD-number directly in the type so distribution matches the data. Migration path: alias fbi.302 → fbi.fd302, decompose fbi.evidence_receipt by form pattern. | 0 |
| 16 | 1778851726 | bulk_reassign | misc/memo/image.scan | fbi.302 / fbi.302a / fbi.evidence_receipt / fbi.ec / fbi.report — by form-number match | Per user insight: mirror FBI form distribution in v2 schema. Bulk-relabeled docs whose v1 was a generic catch-all (image.scan/misc/memo/blank.white) but corpus text contains a specific FD-number. Counts: FD-302a→302a 93, FD-302→302 22, FD-597→evidence_receipt 60, FD-1057→ec 66, FD-1023→report 22, FD-71→ec 15. | 278 |
| 17 | 1778852632 | observation | image.scan (31,715 docs) | DEPRECATE in v2 — every doc fits a more specific type | Subagent #5 reviewed 80 random image.scan docs and 0/80 fit "scanned typed-text". 33% were image.photo (mostly FBI residence walkthrough). Other types found: placeholder.native, placeholder.embedded_image, divider, financial.bank_form, map.diagram, fbi.302a, handwritten. v2 should retire image.scan as a usable type and route every current image.scan doc to its real type. Estimated migration: ~10K → image.photo, ~3K → placeholder.*, etc. | 31715 |
| 18 | 1778852632 | new_type | placeholder.embedded_image | MAPI inline-image text stubs that appear when an email image attachment was extracted as a "page" but the actual image was not produced. Distinct from placeholder.native (DOJ-side) — this is producer/email-system-side. Subagent #5 found 6/80 in image.scan sample. | 0 | |
| 19 | 1778852632 | new_type | financial.bank_form | JPMorgan-style "Print Form — Service Request" structured banking forms. Found 4/80 in image.scan sample + earlier hits in blank. | 0 | |
| 20 | 1778852632 | new_type | image.scan_artifact | Failed/garbled scan output (OCR returned junk, page unreadable, compressed-text rendering). Distinct from blank because there WAS content; the scan failed. | 0 | |
| 21 | 1778852632 | new_type | image.web_screenshot | Screenshot of web content (Google search results, web pages). Found in blank batch. | 0 | |
| 22 | 1778852632 | observation | blank (63,071 docs) + blank.photo (4,288) | Mostly mis-bucketed image.photos — 75/95 in blank sample were small photo centered on white page | v1 blank-detection appears to be pixel-density based (high white-pixel ratio → blank), which mis-classifies any photo with a white background or small photo on white page. Subagent #4 found 75/95 = 79% of "blank" sample docs were actually image.photo. Extrapolating: ~50K of the 63K blank docs may be misclassified image.photo, including a "Caribbean island construction" photo dossier of ~15+ docs in EFTA0113xxxx–0118xxxx range that has investigative value. | 67000 |
| 23 | 1778852632 | producer_observation | Deutsche Bank "Passion to Perform" | divider — recurring producer cover/slip-sheet (~5 in image.scan sample) | Deutsche Bank productions include divider pages bearing the "Passion to Perform" tagline. JPMorgan productions include their own address-block dividers. Worth a producer-id flag for tracking which agency/firm produced each doc. | 0 |
| 24 | 1778852689 | bulk_reassign | misc / blank | placeholder.native (3,859) + divider (1,244) | Two distinct placeholder/slip-sheet patterns hidden in catch-all categories. "No Images Produced" → placeholder.native: DOJ-side native-file placeholder. 3,859 docs, all in v1=misc. "INTENTIONALLY LEFT BLANK" → divider with is_slip_sheet flag: standalone slip sheets (the 1,244 in v1=blank). Did NOT reassign the 1,764 "INTENTIONALLY LEFT BLANK" in financial.bank_statement — those are slip-sheets WITHIN bank statements; keep primary type, but in v2 we could add an is_contains_slip_sheet flag. | 5097 |
| 25 | 1778853213 | observation | memo (9,772 docs) | KEEP as type but acknowledge ~91% of current memo is mislabeled (catch-all) | Subagent #4 sample of 58: only 5 (8.6%) were true internal memos. Top reroutes: letter (28%), FBI form types (22%), financial primitives (17%), image.photo, divider. Like image.scan/blank, memo is a v1 catch-all — but unlike them, "memo" IS a legit content category for the residual ~8%. Keep type, plan v2 migration to route most of v1=memo elsewhere. | 9772 |
| 26 | 1778853426 | observation | financial.equity_swap (4,217 docs) | WRONG name; ~10× more docs than actual equity-swap content. Migration: financial.portfolio_statement family. | Subagent #6 audit of 20 random docs: 0 were actual equity swaps. 14/20 were portfolio/derivatives valuations (esp. Deutsche Bank Southern Financial LLC quarterly MTMs 2013-2015), 4 hedge fund factsheets (Atorus LLC NAV tearsheets), 3 contract/offering docs, 2 emails, 1 BOP memo. v1 keyword-fingerprinted "Equity Option" / "equity swap" / option-pricing terms in derivatives-valuation statements and incorrectly typed them as the swap CONTRACT type. v2: split into financial.portfolio_statement (the … | 4217 |
| 27 | 1778853426 | observation | blank (63K) + misc (17K) bucket combined ~80K | Estimated 60-70K (80%+) are misclassified image.photo | Confirmed across two batches: blank-b1 found 85/100 image.photo; blank-b2 found 75/95 image.photo. Combined batches: 160/195 = 82% are actually photos. Property-photo clusters: Little St. James construction (00542xxx, 0114-0117xxxx), NYC mansion interiors (00000xxx, 0055-0056xxxx). v1 pixel-density classifier broke entirely on photos with white borders. Recommend: per-doc CLIP visual-classifier pass on full 80K, expected ~65K → image.photo migration. | 80000 |
| 28 | 1778853426 | new_type | financial.portfolio_statement | Confirmed as substantial v2 type: Deutsche Bank valuation statements, Morgan Stanley/SoFi/iCapital holdings reports, hedge fund factsheets. Currently scattered across financial.equity_swap, financial.bank_statement, email. | 0 | |
| 29 | 1778853426 | new_type | financial.fund_factsheet | Hedge fund NAV tearsheets (e.g. Atorus LLC). Distinct from portfolio_statement (which is account-level) — these are FUND-level promotional/info sheets. | 0 | |
| 30 | 1778854204 | schema_principle | email.body / email.header / email.chain / email.footer_only as distinct TYPES | single email TYPE + structural flags (is_footer_only, is_thread_chain, ...) + topic.* flags | Per user insight: email sub-types are structural fingerprinting (where the email got cut), not semantic. The atlas already clusters emails by topic via BGE embeddings; v2 mirrors that — one email type, structural state via flag, semantic content via topic.* flag. Subagent #7 sampled 60 v1=email.footer_only and found 98% wrong (53 were actually email.chain via the same structural fingerprinting failure mode). Validates collapse. | 28991 |
| 31 | 1778854625 | atlas_cluster_topic | atlas cluster c50 (26,027 docs) | topic.leon_black_tax_advisory | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 26,027; exemplar v1 types: {'email', 'legal.contract'}. Propagate flag to all cluster members. | 26027 |
| 32 | 1778854625 | atlas_cluster_topic | atlas cluster c49 (20,179 docs) | topic.zorro_ranch_operations | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 20,179; exemplar v1 types: {'email', 'blank.white'}. Propagate flag to all cluster members. | 20179 |
| 33 | 1778854625 | atlas_cluster_topic | atlas cluster c54 (13,240 docs) | topic.macro_market_research | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 13,240; exemplar v1 types: {'email', 'research.report'}. Propagate flag to all cluster members. | 13240 |
| 34 | 1778854625 | atlas_cluster_topic | atlas cluster c7 (30,949 docs) | topic.property_management_nyc | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 30,949; exemplar v1 types: {'plist', 'calendar'}. Propagate flag to all cluster members. | 30949 |
| 35 | 1778854625 | atlas_cluster_topic | atlas cluster c64 (4,440 docs) | topic.att_call_detail_records | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 4,440; exemplar v1 types: {'phone.cdr'}. Propagate flag to all cluster members. | 4440 |
| 36 | 1778854625 | atlas_cluster_topic | atlas cluster c58 (6,903 docs) | topic.travel_itineraries_invoices | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 6,903; exemplar v1 types: {'travel.itinerary', 'financial.invoice'}. Propagate flag to all cluster members. | 6903 |
| 37 | 1778854625 | atlas_cluster_topic | atlas cluster c46 (44,710 docs) | topic.epstein_personal_chat | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 44,710; exemplar v1 types: {'email', 'email.footer_only'}. Propagate flag to all cluster members. | 44710 |
| 38 | 1778854625 | atlas_cluster_topic | atlas cluster c74 (19,516 docs) | topic.news_and_investment_pitches | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 19,516; exemplar v1 types: {'email', 'email.footer_only', 'research.article'}. Propagate flag to all cluster members. | 19516 |
| 39 | 1778854625 | atlas_cluster_topic | atlas cluster c13 (7,066 docs) | placeholder.unexported_attachment | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 7,066; exemplar v1 types: {'misc'}. Propagate flag to all cluster members. | 7066 |
| 40 | 1778854625 | atlas_cluster_topic | atlas cluster c10 (11,653 docs) | producer.jpm | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 11,653; exemplar v1 types: {'travel.itinerary', 'image.scan'}. Propagate flag to all cluster members. | 11653 |
| 41 | 1778854625 | atlas_cluster_topic | atlas cluster c23 (9,759 docs) | producer.deutsche_bank | Atlas-derived natural cluster characterized by subagent vision on 3-4 exemplars. Cluster size 9,759; exemplar v1 types: {'email.footer_only', 'email.body'}. Propagate flag to all cluster members. | 9759 |
| 42 | 1778854625 | new_type | placeholder.unexported_attachment | Atlas cluster c13 (7,066 docs) is USGME load-file placeholders for attachments that DOJ did not actually export. Distinct from placeholder.native (DOJ-side missing-image marker) and placeholder.embedded_image (MAPI inline stubs). This is a significant PRODUCTION GAP finding — references to attachments that should exist but don't. | 7066 | |
| 43 | 1778854960 | atlas_cluster_topic_text | atlas cluster c5 (37,941 docs) | topic.doj_fbi_correspondence | DOJ/FBI/BOP internal correspondence about Epstein case (usdoj.gov + bop.gov + fbi.gov senders; subjects: epstein/maxwell/confidential/request) | 37941 |
| 44 | 1778854960 | atlas_cluster_topic_text | atlas cluster c34 (26,060 docs) | topic.epstein_intellectual_circle | Leon Black / Woody Allen / Leon Botstein scheduling (subjects: leon/woody/botstein/allen/black/dinner; bard.edu sender) | 26060 |
| 45 | 1778854960 | atlas_cluster_topic_text | atlas cluster c63 (36,450 docs) | topic.daily_scheduling | Daily schedule/meeting coordination (lunch/dinner/today/tomorrow/reminder) | 36450 |
| 46 | 1778854960 | atlas_cluster_topic_text | atlas cluster c2 (22,529 docs) | topic.travel_bookings | Travel itinerary / booking confirmations (flight/paris/ticket/itinerary/visa/passport; citicar.com / myamextravel.com) | 22529 |
| 47 | 1778854960 | atlas_cluster_topic_text | atlas cluster c25 (27,850 docs) | topic.southern_financial_llc | Deutsche Bank Southern Financial LLC account thread (db.com sender; subjects: southern/financial/llc/party/edd) | 27850 |
| 48 | 1778854960 | atlas_cluster_topic_text | atlas cluster c22 (25,309 docs) | topic.intellectual_correspondence | Communications with academics — Chomsky, Kuhn, etc. | 25309 |
| 49 | 1778854960 | atlas_cluster_topic_text | atlas cluster c57 (24,320 docs) | topic.social_pr | Peggy Siegal PR/social calendar (peggysiegal.com sender) | 24320 |
| 50 | 1778854960 | atlas_cluster_topic_text | atlas cluster c28 (22,997 docs) | topic.financial_wires | Wire transfers / invoices / financial updates (subjects: invoice/wire/call) | 22997 |