DocumentLens: Extract multimodal data from documents using Llama

Use casesMultimodalRAG

DocumentLens: Extract multimodal data from documents using LlamaBuild a tool to streamline the extraction of text, tables, images, charts into JSON, SQL/vector databases and Excel spreadsheetsRead more

Open on GitHub

Introduction

DocumentLens utilizes Llama to extract structured data from rich documents and store it in various machine-readable formats like JSON, SQL and vector databases. It is designed to handle complex documents with high accuracy and provide flexible configuration options to tailor extraction tasks to specific needs.

Key Features

Text Extraction: Extract and structure main text, titles, captions, etc.
Table Extraction: Convert complex tables into structured data formats.
Image Extraction: Extract images with contextual descriptions and captions.
Chart Extraction: Convert charts and graphs into structured JSON data.
Multiple Output Formats: JSON, CSV, Excel, and SQL database storage.
Vector Search Capabilities: Semantic search across extracted content.

Architecture

The architecture of DocumentLens is modular, comprising several core components that work together to achieve the desired functionality.

Core Components

RequestBuilder: Builds inference requests for LLMs with image and text content.
ArtifactExtractor: Extracts structured data from documents using configurable prompts.
PDFUtils: Handles PDF processing and page extraction as images.
InferenceUtils: Manages LLM inference with support for vLLM and OpenAI-compatible APIs.
JSONUtils: Handles JSON extraction and validation from LLM responses.
ImageUtils: Utility functions for image encoding and processing.

Installation and Setup

To get started with DocumentLens, follow these steps:

1. Clone the repository:

python
git clone https://github.com/meta-llama/llama-cookbook.git
cd llama-cookbook

2. Install project-specific dependencies:

bash
cd end-to-end-use-cases/structured_parser
pip install -r requirements.txt

3. Configure the recipe by editing src/config.yaml to set up your model, API keys, and other parameters.

Usage Examples

Command Line Usage

Extract text from a PDF:

bash
python src/structured_extraction.py path/to/document.pdf text

Extract charts and tables and save them as CSV files:

bash
python src/structured_extraction.py path/to/document.pdf charts,tables --save_tables_as_csv

Process a directory of PDFs and export tables to Excel:

bash
python src/structured_extraction.py path/to/pdf_directory text,tables --export_excel

Code Examples

Extracting Artifacts from a PDF

python
from src.structured_extraction import ArtifactExtractor
from src.utils import PDFUtils
# Extract pages from a PDF
pages = PDFUtils.extract_pages("document.pdf")
# Process specific pages
for page in pages[10:20]:
    artifacts = ArtifactExtractor.from_image(
        page["image_path"],
        ["text", "tables"]
    )
    # Custom processing of artifacts...

Querying the Database

python
from src.json_to_sql import DatabaseManager
# Query all text artifacts
text_df = DatabaseManager.sql_query(
    "sqlite3.db",
    "SELECT * FROM document_artifacts WHERE artifact_type = 'text'"
)
# Query tables containing specific content
revenue_tables = DatabaseManager.sql_query(
    "sqlite3.db",
    "SELECT * FROM document_artifacts WHERE artifact_type = 'table' AND table_info LIKE '%revenue%'"
)

Note: The SQLite3 database engine is bundled with the sqlite3 Python package.

Configuration

The recipe is configured via src/config.yaml. Key configuration options include model configuration, inference parameters, database configuration, and artifact configuration.

Model Configuration

yaml
model:
  backend: openai-compat  # [offline-vllm, openai-compat]
  # For openai-compat
  base_url: "https://api.llama.com/compat/v1"
  api_key: "YOUR_API_KEY"
  model_id: "Llama-4-Maverick-17B-128E-Instruct-FP8"
  # For offline-vllm
  path: "/path/to/checkpoint"
  tensor_parallel_size: 4
  max_model_len: 32000
  max_num_seqs: 32

Extending the recipe

DocumentLens is designed to be extensible. You can add new artifact types, customize extraction logic, and use different models.

Adding New Artifact Types

1. Add a new artifact type configuration in config.yaml:

yaml
   artifacts:
     my_new_artifact:
       prompts:
         system: "Your system prompt here..."
         user: "Your user prompt with {schema} placeholder..."
       output_schema: { # Your JSON schema here }
       use_json_decoding: true

2. Update the CLI, which is implemented in src/structured_extraction.py, to include your new artifact type.

Best Practices

Model Selection: Use larger models for complex documents or when high accuracy is required.
Prompt Engineering: Adjust prompts in config.yaml for your specific document types.
Output Schema: Define precise schemas to guide the model's extraction process.

Troubleshooting

Common issues include model capacity errors, extraction quality issues, and configuration errors. Refer to the Getting Started for more details.

Related GitHub Repositories

ArticleSummarizer - Example implementations of article summarization using Llama models
Contextual-Chunking-RAG - Advanced RAG implementation with contextual chunking strategies
Multimodal-RAG - Multimodal retrieval-augmented generation implementation

Stay up-to-date

Our latest updates delivered to your inbox

Subscribe to our newsletter to keep up with the latest AI updates, releases and more.