Possibility to load documents with Azure Document Intelligence from Azure Container Storage

Question

I have a storage account with Azure Container Storage configured consisting of multiple pdf/word/excel files. I would like to use Azure Document Intelligence to semantically chunk these files.

Is there a possibility to load the files directly from Container Storage to Azure Document Intelligence using langchain? According to the langchain docs it seems like either file has to be locally available or public url has to be handed over.

Attempt:

# Prerequisite: An Azure AI Document Intelligence resource in one of the 3 preview regions: East US, West US2, West Europe

import os
from langchain_community.document_loaders import AzureAIDocumentIntelligenceLoader

file_path = "storage-path-to-file"
endpoint = os.getenv("DOCUMENTINTELLIGENCE_ENDPOINT")
key = os.getenv("DOCUMENTINTELLIGENCE_API_KEY")

loader = AzureAIDocumentIntelligenceLoader(
    api_endpoint=endpoint, api_key=key, file_path=file_path, api_model="prebuilt-layout"
)

documents = loader.load()

# Returns:
# Message: Invalid request.
# Inner error: {
#    "code": "InvalidManagedIdentity",
#    "message": "The managed identity configuration is invalid: Managed identity is not enabled  # for the current resource."
# }

Answer 1

Is there a possibility to load the files directly from Container Storage to Azure Document Intelligence using langchain? According to the langchain docs it seems like either file has to be locally available or public url has to be handed over.

You can use the below code which loads the files directly from Azure Blob storage using Azure Blob URL + SAS token.

Code:

import os
from langchain_community.document_loaders import AzureAIDocumentIntelligenceLoader

#bloburl + ?Sastoken
URL = "https://venkat78932.blob.core.windows.net/test/24005356.pdf?sp=r&st=2024-07-09T12:41:41Z&se=2024-07-09T20:41:41Z&spr=https&sv=2022-11-02&sr=b&sig=eupT8WGH5ojQpXYd%2xxxxxD"
endpoint = os.getenv("DOCUMENTINTELLIGENCE_ENDPOINT")
key = os.getenv("DOCUMENTINTELLIGENCE_API_KEY")


loader = AzureAIDocumentIntelligenceLoader(
    api_endpoint=endpoint, api_key=key, url_path=URL, api_model="prebuilt-layout"
)

documents = loader.load()

print(len(documents))
print(documents[0])

Output:

1
page_content='Word Documents Template\n===\n\n\n## Main heading:\n\nUse the Heading 1 style for primary headings so that screen readers can identify them as such.\n\nIf not already, manually change your heading 1 style to be:\n\n\\- sans serif (e.g. Arial, Verdana, Trebuchet or Calibri),\n\n\\- 16 pt, and\n\n\\- Bold\n\nThen set this formatting as your default for this style.\n\n\n## Sub Headings:\n\nUse Heading 2 style for sub headings.\n\nIf not already, manually change your heading 2 style to be:\n\n\\- sans serif (e.g. Arial, Verdana, Trebuchet or Calibri...................tands out, and does not distort the shape of text as italics and underlining do. Finally, block capitals can be difficult to follow as block capitals remove the natural shape of words, turning them into blocks. Clear layout allows one to focus on the content of visual materials rather than the format.\n\n\n## Furthermore\n\nIf you use headings it makes the creation and upkeep of tables of contents easier (For automatic creation and updating go to: Insert - Reference - Index and Tables - Table of contents).\n'

You can get the Azure Blob URL + SAS token from portal.

Portal -> Storage account -> Container -> your file -> Generate sas token -> click Generate sas token and url

Portal:

Reference: Azure AI Document Intelligence | 🦜️🔗 LangChain