DocumentLens: Extract multimodal data from documents using Llama

Use casesMultimodalRAG

DocumentLens: Extract multimodal data from documents using LlamaBuild a tool to streamline the extraction of text, tables, images, charts into JSON, SQL/vector databases and Excel spreadsheetsRead more

Open on GitHub

Introduction

DocumentLens utilizes Llama to extract structured data from rich documents and store it in various machine-readable formats like JSON, SQL and vector databases. It is designed to handle complex documents with high accuracy and provide flexible configuration options to tailor extraction tasks to specific needs.

Key Features

Text Extraction: Extract and structure main text, titles, captions, etc.

Core Components

RequestBuilder: Builds inference requests for LLMs with image and text content.
ArtifactExtractor: Extracts structured data from documents using configurable prompts.
PDFUtils: Handles PDF processing and page extraction as images.
InferenceUtils: Manages LLM inference with support for vLLM and OpenAI-compatible APIs.
JSONUtils: Handles JSON extraction and validation from LLM responses.
ImageUtils: Utility functions for image encoding and processing.

1. Clone the repository:

python
git clone https://github.com/meta-llama/llama-cookbook.git
cd llama-cookbook

2. Install project-specific dependencies:

bash
cd end-to-end-use-cases/structured_parser
pip install -r requirements.txt

3. Configure the recipe by editing src/config.yaml to set up your model, API keys, and other parameters.

Command Line Usage

Extract text from a PDF:

bash
python src/structured_extraction.py path/to/document.pdf text

Extract charts and tables and save them as CSV files:

bash
python src/structured_extraction.py path/to/document.pdf charts,tables --save_tables_as_csv

Process a directory of PDFs and export tables to Excel:

bash
python src/structured_extraction.py path/to/pdf_directory text,tables --export_excel

Code Examples

Extracting Artifacts from a PDF

python
from src.structured_extraction import ArtifactExtractor
from src.utils import PDFUtils
# Extract pages from a PDF
pages = PDFUtils.extract_pages("document.pdf")
# Process specific pages
for page in pages[10:20]:
    artifacts = ArtifactExtractor.from_image(
        page["image_path"],
        ["text", "tables"]
    )
    # Custom processing of artifacts...

Querying the Database

python
from src.json_to_sql import DatabaseManager
# Query all text artifacts
text_df = DatabaseManager.sql_query(
    "sqlite3.db",
    "SELECT * FROM document_artifacts WHERE artifact_type = 'text'"
)
# Query tables containing specific content
revenue_tables = DatabaseManager.sql_query(
    "sqlite3.db",
    "SELECT * FROM document_artifacts WHERE artifact_type = 'table' AND table_info LIKE '%revenue%'"
)

Note: The SQLite3 database engine is bundled with the sqlite3 Python package.

Configuration

The recipe is configured via src/config.yaml. Key configuration options include model configuration, inference parameters, database configuration, and artifact configuration.

Model Configuration

yaml
model:
  backend: openai-compat  # [offline-vllm, openai-compat]
  # For openai-compat
  base_url: "https://api.llama.com/compat/v1"
  api_key: "YOUR_API_KEY"
  model_id: "Llama-4-Maverick-17B-128E-Instruct-FP8"
  # For offline-vllm
  path: "/path/to/checkpoint"
  tensor_parallel_size: 4
  max_model_len: 32000
  max_num_seqs: 32

Extending the recipe

DocumentLens is designed to be extensible. You can add new artifact types, customize extraction logic, and use different models.

Adding New Artifact Types

1. Add a new artifact type configuration in config.yaml:

yaml
   artifacts:
     my_new_artifact:
       prompts:
         system: "Your system prompt here..."
         user: "Your user prompt with {schema} placeholder..."
       output_schema: { # Your JSON schema here }
       use_json_decoding: true

2. Update the CLI, which is implemented in src/structured_extraction.py, to include your new artifact type.

Best Practices

Model Selection: Use larger models for complex documents or when high accuracy is required.
Prompt Engineering: Adjust prompts in config.yaml for your specific document types.
Output Schema: Define precise schemas to guide the model's extraction process.

Related GitHub Repositories

ArticleSummarizer - Example implementations of article summarization using Llama models
Contextual-Chunking-RAG - Advanced RAG implementation with contextual chunking strategies
Multimodal-RAG - Multimodal retrieval-augmented generation implementation

Please confirm your identity