Building a Searchable Archive: How AI Processes 8,000+ Legal Documents

Court documents, depositions, financial records, and legal filings are the raw material of transparency in democratic societies. When high-profile legal cases conclude or when freedom of information requests are fulfilled, thousands of pages are often released to the public. But there's a problem: these documents are almost always scanned images, not searchable text.

A scanned PDF is, to a computer, just a collection of pixels a photograph of a page. You cannot search for a person's name. You cannot extract dates. You cannot cross-reference mentions of a company across hundreds of documents. For journalists, researchers, and concerned citizens, this creates an insurmountable barrier. Reading 8,000 pages manually is impossible for a single person and impractical even for a team.

The answer lies in Optical Character Recognition (OCR) powered by modern AI vision models. But OCR alone is not enough. To make a document archive truly useful, you need entity extraction, deduplication, document reconstruction, and semantic search. This is the story of how one open-source project solved all of these problems.

Traditional OCR tools like Tesseract work well for clean, printed text. But legal documents are rarely clean. They contain:

• • Handwritten annotations in margins and on forms
• • Stamps and signatures overlaid on text
• • Multi-column layouts that confuse reading order
• • Redaction boxes covering sensitive information
• • Low-resolution scans from older fax or photocopy systems

The solution: AI-powered computer vision models from OpenAI, Google, or similar providers. These models, originally designed for image understanding, excel at extracting text from complex documents. The project used a Python script (process_images.py) that:

The script supports parallel processing with configurable workers (default: 5 concurrent requests) and can be limited to process only the first N images for testing. Failed files are logged for later retry using a cleanup script.

OCR gives you the text. But to make it searchable by entity, you need Named Entity Recognition (NER). The AI vision model extracts entities directly from the document images during OCR processing:

These entities are stored alongside the OCR text in each document's JSON file. The vision model also extracts dates (11,020 temporal references) and document types (41 categories like "Deposition", "Flight Log", "Email", etc.).

But there's a critical problem that emerges at scale: inconsistent naming.

When processing thousands of pages, the AI vision model does not maintain consistency across documents. One page might extract "Epstein", another "Jeffrey Epstein", and a third "J. Epstein". Without deduplication, these would appear as three separate people in the index.

The solution is a second AI pass dedicated to entity resolution. The script deduplicate.py collects all extracted entities, groups them in batches (default: 50 entities per batch), and sends them to the AI with instructions to identify duplicates and provide a canonical name for each cluster.

4.1 How Deduplication Works

The same process runs for document types (deduplicate_types.py), merging variants like "deposition", "Deposition", and "DEPOSITION TRANSCRIPT" into a single canonical form. The result: 41 document type categories instead of potentially hundreds of variations.

Legal document releases typically arrive as a folder of individual page scans: page_001.jpg, page_002.jpg, and so on. But each document spans multiple pages. A deposition might be 50 pages. A financial record might be 200.

During OCR processing, the AI extracts document numbers and page numbers from each page (if present). The build system uses these metadata fields to:

• • Group pages by document number
• • Sort pages within each document by page number
• • Reconstruct complete multi-page documents for display

This is non-trivial because document numbers are not consistently formatted. Some pages show "Doc 123", others show "Document #123", and still others show "EPSTEIN-000123". The AI model is instructed to normalize these into a consistent format during extraction.

Having searchable text and entity indexes is powerful. But for a researcher trying to understand which documents are important, reading through hundreds of depositions is still overwhelming. The final layer of the pipeline adds AI-generated document analysis.

The script analyze_documents.py groups pages into complete documents (using the reconstruction logic) and sends the full text of each document to an AI model with instructions to generate:

All analyses are saved to analyses.json and automatically incorporated into the searchable website. The script is resume-friendly it skips documents already analyzed unless forced to re-run.

{
  "document_type": "Deposition",
  "key_topics": [
    "Flight logs",
    "Private aircraft",
    "Passenger manifests"
  ],
  "key_people": [
    {
      "name": "Jeffrey Epstein",
      "role": "Aircraft owner"
    }
  ],
  "significance": "Documents flight records showing...",
  "summary": "This deposition contains testimony regarding..."
}

The entire system is built on open-source tools and is itself open-source. Here's the technology breakdown:

Processing 8,000+ documents with AI is not instant. Here's what the performance profile looks like:

The system is resume-friendly. If processing is interrupted (network failure, rate limit, manual stop), the scripts automatically skip already-processed files on the next run. This is critical for large corpora where processing might take days.

This project is a case study in how AI can democratize access to information that was previously locked away. Before this archive existed, these documents were technically "public" but practically inaccessible. A journalist could spend weeks reading through scans with no ability to search across documents. A researcher trying to map networks of individuals would have to manually cross-reference hundreds of pages.

Now, anyone with a web browser can:

• • Search for any person, organization, or location across all documents
• • Read AI-generated summaries to identify relevant documents quickly
• • Browse by date, document type, or entity
• • Copy and quote exact text passages for analysis or reporting

The project maintainers have outlined an ambitious next phase: relationship graphs. Instead of just listing entities, the system could visualize connections between people, organizations, and locations based on co-occurrence in documents.

Implementation would use client-side JavaScript graph libraries like D3.js or Cytoscape.js to render interactive visualizations. The data would be pre-generated during the build process and served as static JSON files, maintaining the project's philosophy of simple, fast, host-anywhere static sites.

The deduplication step becomes even more critical here without it, "Epstein" and "Jeffrey Epstein" would appear as separate, disconnected nodes in the graph.

This project raises important questions about the role of AI in public interest technology. On one hand, AI-powered OCR and entity extraction dramatically reduce the barrier to accessing public records. On the other hand, there are legitimate concerns:

That said, the transparency of the process is a key strength. The code is open source. The prompts are visible. The raw OCR output (full JSON files) can be examined. And most importantly, the original document images are always linked users can verify any claim against the source.

This project is more than a technical demonstration. It's a proof of concept for a new model of legal transparency. In an era where courts are increasingly releasing documents as scanned PDFs, the ability to automatically process, index, and analyze these documents at scale is transformative.

The tools are accessible: Python, OpenAI APIs, static site generators. The cost is manageable: hundreds of dollars, not tens of thousands. The results are profound: thousands of documents that would have remained practically inaccessible are now searchable by anyone with an internet connection.

For developers, this is a template. The code is open source. You can fork it, adapt it, and apply it to any document corpus you care about. For journalists and researchers, it's a reminder that the tools to hold power accountable are getting better, faster, and more accessible.