MARS Technical Architecture

System Overview

The MARS architecture follows a two-phase approach: first building a reference knowledge base (exemplars), then using that knowledge to classify and recover data from damaged or carved databases (candidates).

High-Level Architecture

┌─────────────────────────────────────────────────────────┐
│                   EXEMPLAR PHASE                        │
│  (Known-Good macOS System → Reference Knowledge Base)   │
└─────────────────────────────────────────────────────────┘
                           │
                           ▼
         ┌──────────────────────────────────┐
         │   Scan Live System / Image       │
         │   Extract Database Schemas       │
         └──────────────────────────────────┘
                           │
                           ▼
         ┌──────────────────────────────────┐
         │   Generate Rubrics (JSON)        │
         │   Create Hash Lookup Table       │
         └──────────────────────────────────┘

┌─────────────────────────────────────────────────────────┐
│                   CANDIDATE PHASE                       │
│     (Carved Files → Classified & Recovered Data)        │
└─────────────────────────────────────────────────────────┘
                           │
                           ▼
         ┌──────────────────────────────────┐
         │   File Categorization            │
         │   (raw_scanner/file_categorizer) │
         └──────────────────────────────────┘
                           │
                           ▼
         ┌──────────────────────────────────┐
         │   Database Variant Selection     │
         │   (O/C/R/D/X variant testing)    │
         └──────────────────────────────────┘
                           │
                ┌──────────┴──────────┐
                │                     │
                ▼                     ▼
    ┌──────────────────┐   ┌──────────────────┐
    │  Variant X       │   │  Valid Variants  │
    │  (Empty/Failed)  │   │  (O/C/R/D)       │
    └──────────────────┘   └──────────────────┘
                │                     │
                ▼                     ▼
    ┌──────────────────┐   ┌──────────────────┐
    │  Byte Carving    │   │  LF Processor    │
    │  (Extract raw)   │   │  (4 modes)       │
    └──────────────────┘   └──────────────────┘
                │                     │
                └──────────┬──────────┘
                           ▼
         ┌──────────────────────────────────┐
         │   Final Output Organization      │
         │   (catalog/, metamatches/, etc.) │
         └──────────────────────────────────┘

Data Flow: Exemplar → Candidate

The system operates in two phases:

Key Terminology

Exemplar Scanning

Exemplar scanning builds the reference knowledge base that powers the entire recovery pipeline. By analyzing known-good databases from a live macOS system, MARS creates detailed schema fingerprints that enable accurate classification and reconstruction of carved data.

What Gets Extracted

The exemplar scanner performs comprehensive analysis of each database:

• Schema Structure: Full table and column definitions
• Column Roles: Semantic classification (timestamp, UUID, URL, email, etc.)
• String Statistics: Average string lengths for disambiguation
• Example Values: Sample data for pattern matching
• Row Counts: Expected data volumes per table

Rubric Generation Process

Hash-Based Fast Matching

The hash lookup table enables rapid O(1) exemplar identification without loading full rubrics or iterating through databases:

MD5(table1|col1,col2,col3|table2|col4,col5)

{
  "a1b2c3d4": "Safari_History",
  "e5f6g7h8": "Chrome_Cookies",
  "i9j0k1l2": "Messages_chat"
}

Rubric System

Rubrics are the intelligence layer that enables MARS to match fragmented data back to its original table structure.

Rubric Anatomy

A complete rubric contains three layers of information:

Column Roles & Semantic Anchors

Semantic anchors are weighted scores that boost confidence when specific patterns are detected. They help create a strong "fingerprint" and disambiguate between similar schemas:

String Statistics for Disambiguation

Average string lengths (avg_length) help distinguish between similar tables with different purposes:

Rubric Matching Algorithm

When matching lost_and_found fragments to rubrics, MARS uses a multi-factor confidence scoring system:

1. Column Matching: Compare column names and types between fragment and rubric
2. Chunk Analysis: Identify longest matching column sequences (chunks)
3. Semantic Boost: Add weights for detected semantic anchors
4. String Validation: Compare avg_length statistics
5. Threshold Filtering: Reject matches below minimum confidence (default 0.7)

Candidate Scan Pipeline

The candidate pipeline processes carved files through multiple specialized stages, each handling a specific aspect of recovery and classification.

Pipeline Stages

1. raw_scanner/         File Discovery & Categorization
   file_categorizer     ├─ Identify file types via fingerprinting
                        ├─ Skip media/executables
                        └─ Organize by artifact type

                              │
                              ▼

2. fingerprinter/       Type-Specific Identification
                        ├─ SQLite: Magic bytes + schema extraction
                        ├─ Text logs: Pattern-based classification
                        └─ Archives: Compression format detection

                              │
                              ▼

3. matcher/             Schema Matching & Classification
   rubric_matcher       ├─ Hash-based O(1) exemplar lookup
                        ├─ Rubric confidence scoring
                        └─ Group unmatched by schema similarity

                              │
                              ▼

4. db_variant_selector/ Recovery Variant Testing
                        ├─ O (Original): Test as-is
                        ├─ C (Clone): Clean copy via VACUUM
                        ├─ R (Recover): SQLite .recover output
                        ├─ D (Dissect): sqlite_dissect rebuild
                        └─ X (Failed): Send to byte carving

                              │
                ┌─────────────┴─────────────┐
                │                           │
                ▼                           ▼

5a. carver/           5b. lf_processor/
    Byte-Level Carving      Fragment Reconstruction
    ├─ Extract timestamps   ├─ MERGE: Metamatch groups
    ├─ Parse protobuf       ├─ CATALOG: Exact matches
    ├─ Analyze URLs         ├─ NEAREST: Best-fit exemplar
    └─ Output JSONL/CSV     └─ ORPHAN: Unmatched

                              │
                              ▼

6. output/              Final Organization
   structure            ├─ catalog/ (exact matches)
                        ├─ metamatches/ (identical schemas, no exemplar)
                        ├─ found_data/ (L&F orphans)
                        └─ carved/ (byte-carved)

Stage 1: File Categorization

The file categorizer scans input directories and identifies file types using magic bytes and content analysis:

• SQLite databases: Identified by SQLite format 3\x00 magic bytes
• Text logs: WiFi logs, system logs, install logs via pattern matching
• Archives: gzip, bzip2 (decompressed and re-scanned)
• Ignored: Images, videos, executables (defined in config)

Stage 2: Fingerprinting

The fingerprinter performs deep type analysis beyond simple magic bytes:

Stage 3: Schema Matching

The matcher uses the exemplar hash lookup table for instant classification:

1. Compute schema hash of candidate database
2. O(1) lookup in exemplar_hash_lookup.json
3. On match: Load full rubric for detailed validation
4. On miss: Group with other unmatched databases by schema hash

Stage 4: Database Variant Selection

See detailed section below on variant selection system.

Stage 5: Lost & Found Reconstruction

See detailed section below on LF processing (MERGE/CATALOG/NEAREST/ORPHAN).

Stage 6: Output Organization

Final databases are organized by match quality and processing path:

Database Variant Selection System

When a carved SQLite database is encountered, it may be corrupted, incomplete, or structurally damaged. The variant selector attempts multiple recovery strategies and chooses the best result.

The O/C/R/D/X Variant Approach

Each candidate database is processed through up to five different recovery methods:

Variant Selection Logic

Variant Scoring Heuristics

Each variant is scored based on multiple factors:

• Integrity: PRAGMA integrity_check result (pass = higher score)
• Row Count: Total rows across all tables (more = better)
• Table Completeness: Percentage of expected tables present
• Match Quality: Exact hash match vs. fuzzy table match
• lost_and_found Presence: R variant bonus for LF reconstruction potential

{
  "case_name": "f12345678",
  "chosen_variant": "R",
  "match_type": "hash",
  "exemplar_name": "Safari_History",
  "profile_score": 0.95,
  "variants": {
    "O": {"valid": true, "rows": 150, "integrity": "ok"},
    "C": {"valid": true, "rows": 150, "integrity": "ok"},
    "R": {"valid": true, "rows": 168, "has_lf": true},
    "D": {"valid": false}
  }
}

Residue Processing

After variant selection, the residue processor performs cleanup and extraction:

1. Lost-and-Found Extraction: Extract lost_and_found_* tables from chosen variant into separate databases
2. Storage Cleanup: Delete non-chosen variants to save disk space

Lost & Found (LF) Reconstruction

When SQLite's .recover command succeeds, it creates lost_and_found tables containing orphaned database pages. The LF processor matches these fragments against exemplar rubrics and reconstructs coherent databases.

The Four Processing Modes

LF reconstruction follows a prioritized processing order based on match quality:

Processing Order & Rationale

MERGE: Metamatch Groups (Identical Schemas)

When multiple databases share identical schemas (same tables AND columns) but don't match any known exemplar, they're grouped by schema hash and processed together:

1. Grouping: Databases classified by schema hash (tables + columns)
2. Combining: Merge all group members into single database
3. Superrubric Generation: Create schema rubric from merged data
4. Fragment Matching: Match lost_and_found tables against superrubric
5. Reconstruction: Rebuild combined database with intact + recovered data

CATALOG: Exact Matches

Exact schema matches use the canonical exemplar rubric for reconstruction:

1. Hash Match: Schema hash exactly matches exemplar
2. Rubric Loading: Load canonical exemplar rubric
3. Fragment Matching: Match lost_and_found columns to rubric tables
4. Reconstruction: Rebuild using exemplar schema as template
5. Remnant Handling: Unmatched fragments saved to lost_and_found/

NEAREST: Best-Fit Exemplar Matching

When a database doesn't exactly match any exemplar but is close enough for useful reconstruction, NEAREST matches it to the nearest exemplar schema based on similarity:

1. Schema Comparison: Find the nearest matching exemplar based on table and column similarity
2. Fragment Matching: Match lost_and_found fragments against the nearest exemplar rubric
3. Schema Rebuild: Reconstruct database using the nearest exemplar schema as template (structurally compatible with CATALOG results)
4. Initial Output: Results go to found_data/ initially
5. Phase 7 Reclassification: Successful recovery (L&F rows > 0) promotes to catalog/; no recovery moves to empty/

ORPHAN: Unmatched Fragments

Fragments that don't match any schema are preserved for manual review:

• Collection: Gather all remnants from MERGE/CATALOG/NEAREST processing
• Preservation: Create standalone databases with original fragment structure
• Naming: Include match hints when partial matches existed
• Traceability: Filenames preserved for forensic correlation

Data Source Tracking

All reconstructed tables include a data_source column for provenance:

Manifest Files

Each reconstructed database includes a *_manifest.json documenting:

• Source databases that contributed data
• Intact rows vs. LF-recovered rows per source
• Remnant tables (unmatched fragments)
• Duplicates removed during deduplication
• Table-level statistics (row counts per table)

Byte-Carving Pipeline

When all variant recovery methods fail (variant X), MARS falls back to byte-level carving. This extracts raw data directly from database pages without relying on SQLite's structural integrity.

Carving Strategy

The byte carver processes databases page-by-page, extracting forensic artifacts:

Carving Process Flow

Database File (Variant X)
        │
        ▼
┌───────────────────┐
│ Read Page-by-Page │ (4096-byte SQLite pages)
└───────────────────┘
        │
        ├─── Scan for numeric values ───→ Timestamp Classifier
        │                                  ├─ Unix epoch?
        │                                  ├─ Cocoa timestamp?
        │                                  ├─ Chrome time?
        │                                  └─ Valid range filter
        │
        ├─── Regex match URLs ──────────→ Unfurl Analyzer
        │                                  ├─ Parse structure
        │                                  ├─ Extract query params
        │                                  └─ Detect embedded timestamps
        │
        ├─── Scan for BLOB data ────────→ Protobuf Decoder
        │                                  ├─ blackboxprotobuf decode
        │                                  ├─ Extract nested data
        │                                  └─ Convert to JSON
        │
        └─── Extract text strings ──────→ Text Scanner
                                           ├─ Printable ASCII filter
                                           └─ Context preservation
        │
        ▼
┌──────────────────────────────────┐
│ Output Generation                │
│ ├─ timestamps.csv (optional)     │
│ ├─ carved.jsonl (detailed)       │
│ └─ carved.db (structured SQLite) │
└──────────────────────────────────┘

Integration with Unfurl

Unfurl provides context-aware URL analysis that helps distinguish real timestamps from ID values:

https://facebook.com/photo.php?fbid=123456789&id=987654321

Integration with blackboxprotobuf

Schema-agnostic protobuf decoding recovers structured data without requiring .proto definitions:

Input: Binary BLOB (unknown structure)

blackboxprotobuf output:
{
  "message": {
    "1": "user@example.com",
    "2": 1705584600,
    "3": {
      "1": "https://example.com/path",
      "2": 42
    }
  },
  "typedef": {
    "1": {"type": "string"},
    "2": {"type": "int"},
    "3": {"type": "message", ...}
  }
}

Carving Output Formats

dfVFS Integration

MARS uses Digital Forensics Virtual File System (dfVFS) for universal disk image access. This provides consistent file access across all forensic image formats and archive types.

Supported Formats

Glob Pattern Matching

MARS extends dfVFS with full globstar (**) support for flexible pattern matching:

Globstar Implementation

The ** wildcard matches zero or more directory levels:

• ** at start: Matches from root (any depth prefix)
• /**/ in middle: Matches zero or more intermediate directories
• /** at end: Matches everything below current level
• * alone: Matches exactly one directory level

Volume System Integration

dfVFS provides automatic volume enumeration and metadata extraction:

• GPT Partitions: Partition name, GUID, size, type
• APFS Containers: Volume names, encryption status, roles
• Volume Labels: Filesystem labels from volume attributes

EWF/E01 Mount Utilities

MARS includes utilities for mounting forensic images for interactive exploration:

• macOS: Requires Fuse-T for userspace filesystem mounting
• Windows: Arsenal Image Mounter or similar tools
• Linux: libewf + FUSE for native EWF support

Directory Filtering & Exclusion

To prevent hangs on directories with millions of cache files, MARS includes smart exclusions:

Configuration System

MARS uses a centralized configuration system defined in config/schema.py. All settings are organized into logical sections with dataclasses.

Configuration Sections

Ignorable Tables

GLOBAL_IGNORABLE_TABLES defines tables that are always filtered during schema comparison:

User-Configurable Settings

Settings marked with user_configurable=True are exposed in the UI and saved to .marsproj files:

• Basic: Debug mode, progress bars
• Exemplar: Date ranges, catalog groups, excluded file types
• Carver: Timestamp filtering, protobuf decoding, CSV export
• Advanced: Variant selection, dissect options