RAG with Python: Session Resources

[multispark]

This is the resources and discussion thread for this live stream: https://aka.ms/raghack/python
It will be updated with links after the session.

Recording: https://www.youtube.com/watch?v=HQzNw3j23vE
Slides:
PDF- https://speakerdeck.com/pamelafox/raghack-building-rag-apps-in-python
PPT- https://github.com/microsoft/RAG_Hack/blob/main/slides/RAGHack-RAGInPython.pptx

Code samples:

• RAG on CSV
• RAG on AI Search
• End-to-end RAG chat solution with AI Search

Please ask any follow-up questions here, and the speakers will see them.

[zzulueta]

Questions:

1. In your Jupyter notebook (16:00 of video), the Azure AI Search index was already created. How do we manually create the Index from markdown files using Python?
2. Can we have a copy of the Jupyter notebook?
3. In RAG, the user prompt plus the content is the new prompt. When the content is huge, the token cost for the input also bloats up. This is a concern by customers. How do you count the input token and how do we limit the size?

[pamelafox]

1. This example shows how to create a basic index: https://github.com/pamelafox/vector-search-demos/blob/main/azure_ai_search.ipynb
   The large index from that example was created by this code: https://github.com/Azure-Samples/azure-search-openai-demo/blob/9073b65bc86dcffaba5a6c8d332b14e0102912b7/app/backend/prepdocslib/searchmanager.py#L71
2. Yep! Grab notebooks from:
   https://github.com/pamelafox/vector-search-demos
   and https://github.com/pamelafox/python-openai-demos
3. We count the tokens, using a package that I maintain that uses tiktoken behind the scenes. I discuss it in this blog post: https://blog.pamelafox.org/2024/06/truncating-conversation-history-for.html
   If there are too many messages in history, then I only add the most recent, until I reach the token limit.

Good questions!

[zzulueta]

Thanks for the replies!

So, if I already have an index created using the Azure AI Search Import and Vectorize Data option, I can use your "Search a larger index" code in the notebook to search that index?

On a separate note, I have an unrelated question:

1. Azure AI Studio allows you to create a RAG model using Azure Prompt Flow. When deploying a model using Azure Prompt Flow there is an option there to choose which servers and the number of servers to use for deployment. This requires a monthly upkeep as the servers are running 24/7.
2. However, if you create python code to simulate the RAG of Azure AI Studio and encapsulate it in an API like Azure Functions, it becomes serverless and free for the first 1Mil calls. This approach seems to be cheaper especially for smaller clients who would not exceed 1Mil calls.

Option 2 seems to be the cheaper approach. Is the RAG website that we have in the repository running under a deployed or serverless costing model?

[pamelafox]

Yes, you should be able to use that code, as long as you set up with SearchClient with the correct credential or key, and you specify the name of the embedding field to match your index.

The azure-search-openai-demo repo uses App Service, which does have a free tier option. There's also a branch that was opened this week for optional deployment to Azure Container Apps, which has serverless-like pricing.

The JS session presented their serverless template that does run entirely on Azure Functions: https://github.com/Azure-Samples/serverless-chat-langchainjs

[namtran6701]

I was going to ask this in the session but it ran out of time. Are there any recommended evaluation frameworks that we should use to evaluate our model?

[pamelafox]

I'll discuss that in a session next week:
https://reactor.microsoft.com/reactor/events/23414/
and have a blog post about my evaluation approach here:
https://blog.pamelafox.org/2024/01/evaluating-rag-chat-app-approach-sdks.html

Besides Microsoft's promptflow-evals SDK, you can also try DeepEval, RAGAS, Langchain, etc.

[learnsol]

i am having trouble with the authentication . i cant seem to fix it. i am not an expert in this
code:
import csv
import os
import azure.identity
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
import openai
from dotenv import load_dotenv
import logging

Setup the OpenAI client to use either Azure, OpenAI.com, or Ollama API

load_dotenv(override=True)
API_HOST = os.getenv("API_HOST")

if API_HOST == "azure":
token_provider = azure.identity.get_bearer_token_provider(
azure.identity.DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default"
)
client = openai.AzureOpenAI(
api_version=os.getenv("AZURE_OPENAI_VERSION"),
azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
azure_ad_token_provider=token_provider
)
MODEL_NAME = os.getenv("AZURE_OPENAI_DEPLOYMENT")

error:
openai.AuthenticationError: Error code: 401 - {'statusCode': 401, 'message': 'Unauthorized. Access token is missing, invalid, audience is incorrect (urn:ms.scopedToken or urn:ms.faceSessionToken), or have expired.'}

[pamelafox]

Which repo is this with? Are you trying to use with Azure OpenAI or OpenAi.com or Ollama?

[learnsol]

I used the python openai demos repo , the code that is executed is the retrieval augmented generation.py.
I am using azure open ai.

I have a personal PAYG Azure subscription and logged in with my id which is the owner of the RG which hosts the azure openai resource.

[pamelafox]

Did you set up keyless authentication for your Azure OpenAI account? That code assumes keyless access, but you need to give your user access in that case. I show how to setup keyless access in this blog post: https://blog.pamelafox.org/2023/09/best-practices-for-openai-chat-apps-go.html

[pamelafox]

We also discuss it in this video: https://www.youtube.com/watch?v=IkDcQvKoQ8k

[learnsol]

yes got it working. cognitive services openai user role permissions were missing . thank you for the video link.