Python + AI livestream series: Resources

[pamelafox]

Join us for our 9-part live stream series on using Python with Generative AI models! 🐍 🤖

Register for the series

Livestreams

The livestreams are over, but you can catch up with recordings here:

Topic	Slides	Recording
LLMs	https://aka.ms/pythonai/slides/llms	https://aka.ms/PythonAI107-f
Vector Embeddings	https://aka.ms/pythonai/slides/embeddings	https://aka.ms/PythonAI108-f
RAG	https://aka.ms/pythonai/slides/rag	https://aka.ms/PythonAI109-f
Vision Models	https://aka.ms/pythonai/slides/vision	https://aka.ms/PythonAI1014-f
Structured Outputs	https://aka.ms/pythonai/slides/structuredoutputs	https://aka.ms/PythonAI1015-f
AI Quality & Safety	https://aka.ms/pythonai/slides/qualitysafety	https://aka.ms/PythonAI1016-f
Tool Calling	https://aka.ms/pythonai/slides/toolcalling	https://aka.ms/PythonAI1021-f
AI Agents	https://aka.ms/pythonai/slides/agents	https://aka.ms/PythonAI1022-f
Model Context Protocol (MCP)	https://aka.ms/pythonai/slides/mcp	https://aka.ms/PythonAI1023-f

Office hours

Each Thursday at 1PM PT, we'll have an office hours in the Azure AI Discord. Bring your questions there or post them here!

Code samples

These are the repositories we demonstrated during the series.

Getting started (with free GitHub Models support):

• https://github.com/Azure-Samples/python-openai-demos
• https://github.com/Azure-Samples/openai-chat-vision-quickstart
• https://github.com/Azure-Samples/vector-embeddings-demos
• https://github.com/Azure-Samples/azure-openai-entity-extraction
• https://github.com/Azure-Samples/ai-quality-safety-demos
• https://github.com/Azure-Samples/python-ai-agent-frameworks-demos/
• https://github.com/Azure-Samples/python-mcp-demo

End-to-end solutions (with deployment to Azure):

• https://github.com/Azure-Samples/azure-search-openai-demo
• https://github.com/Azure-Samples/rag-postgres-openai-python
• https://github.com/Azure-Samples/image-search-aisearch
• https://github.com/Azure-Samples/rag-with-azure-ai-search-notebooks/
• https://github.com/pamelafox/issue-triager-agent

Related series

If you are enjoying this series, you might also like:

• RAG Deep Dive, January 2025
• Securing AI Apps on Azure, July 2024
• Python + AI, March 2025

[amynic]

Thanks for sharing with this community @pamelafox - looking forward to the series in October

[bmerkle]

@pamelafox thanks for the cool session today. I have a short question:

• ​​is there any framework/tools support to effective conduct this evaluation (reduce quantization for a vector db) on a large dataset ?
• meaning i want to evaluate how far i can go with quantization with my dataset without loosing quality

[pamelafox]

@bmerkle To evaluate retrieval results, you would need ground truth data representing the correct results. You could construct that ground truth based off an exhaustive search.
Then you would retrieve with the different settings and compute various metrics (like NDCG, recall, etc) to compare the results:
Here's a blog post where someone did accuracy and speed benchmarking for Postgres:
https://tiffena.me/posts/benchmark-chroma-postgres/
Scroll down to "accuracy" to see the metric calculations.

If you're using AI Search, here's how I constructed multiple indexes with different compression parameters:
https://github.com/microsoft/rag-time/tree/main/Journey%203%20-%20Optimize%20your%20Vector%20Index%20for%20Scale
(I did not run the formal retrieval metrics there though- that'd be the next step).

[NnamdiOdozi]

Hi Pam

I just wondered if it wouldn't be a good idea to also link the URL for the version of the course from earlier in the year - I know that there is a Youtube playlist for it. Helps people who like your stuff but keen to get ahead.

[pamelafox]

Sure, I've added the link, good idea.

[wigging]

Will this series talk about how to deploy RAG applications? I am enjoying the series but so far there has not been any discussion about how to deploy AI based web apps. For example, if I have a RAG application that uses a ChromaDB vector database, a FastAPI server, and some kind of web app frontend, then how would I deploy such an app? If this series doesn't talk about deployment, are there any references that you can share to help me deploy my RAG app?

[pamelafox]

Are you asking what Azure services I'd recommend? I deploy the majority of mine on Azure Container Apps. If you look at my GitHub profile, you can CTRL+F for FastAPI and see various examples that have infra included:
https://github.com/pamelafox/
Probably the closest to your setup is this RAG with FastAPI and Postgres:
https://github.com/Azure-Samples/rag-postgres-openai-python/
I haven't used ChromaDB in production, so I can't make a recommendation there.

I tend to serve both the frontend and the backend from the same container, but you could also use separate containers, OR use container routing rules, such that you could have different caching and customizations on frontend vs backend.

Let me know if you have specific questions that I can answer more here. Generally we're not spending much time on prod deployment in this series, but most of the E2E samples shown do have infrastructure-as-code included.

[wigging]

What are the pros/cons of deploying in Docker containers instead of on a single Linux virtual machine? It seems like you could run the frontend and backend all on one Linux server. This would get rid of all the Docker config files and dealing with port mapping between all the containers. Using Docker containers feels like a bunch of extra work when I can just run everything on one Linux VM.

[pamelafox]

Hm, good question. How would the Linux VM deal with scaling? With Container Apps, I can set up auto-scaling rules based off incoming HTTP traffic, to go from 0 or 1 replicas to any max number I configure. That seems like a big benefit to me.

[wigging]

You can use things like Nginx (or Caddy), Uvicorn workers, async database clients, and async applications to handle the load on a single virtual machine. An 8 core VM should be able to handle traffic for most projects with thousands of requests per second. Of course it depends on what your app does but VM hardware is more than adequate these days to handle most projects.

[AndrewMichael2020]

Structured outputs is probably one of the most useful use of LLMs for me that I recently thought of / discovered for myself, and I call it "creating complexity from chaos".

I played with this idea and made a prototype for an app that reads a collection of news and images about say a mountaineering accident, then extracts details from each news source into a single event json document, and then writes a report about the event: https://accident-reports-frontend-5t5l4f4etq-uw.a.run.app/

Caveats for things I needed:

• when extracting, I need a metric or metrics around translating text into a json document value (confidence? something; and some form of aggregation of that metric when jsons fields are combined or merged); hope to think more about this during the "AI Quality & Safety" session;
• grounding in the text; and
• a judges somewhere in the pipeline to evaluate that what I have extracted actually makes sense when taking the whole news story (input) as a context.

[NnamdiOdozi]

I'm guessing the benefit of an LLM doing structured output is that you don't need to do it yourself manaullay or using a tool like LangExtract. I'm thinking it also reduces hallucination and if you use Pydantic baseclass then you can have scheme enforcement in the LLM's output

[bmerkle]

@pamelafox can we please add this URL https://github.com/Azure-Samples/rag-with-azure-ai-search-notebooks/ in the table above
as it is used in your RAG video / slide on page 32.
Thanks so much for the cool sessions :-)

[pamelafox]

Sure added!

[CharlesDraconian]

Small thing but in the 7th session I think links are mixed up.

[pamelafox]

Fixed thank you!

[CharlesDraconian]

Thanks!!! Great content

[NnamdiOdozi]

So I had a question from today's Agents session. When using short term memory, you can summarise the messages/info and stick into the context as a message but i'm guessing the original messages don't get lost but remain in memory, maybe a cache or somewhere? With long-term memory, both summary and original probably go into some DB

[pamelafox]

That's a great question, it depends on how you decide to implement summarization.
For example, langchain offers this summarization middleware:
https://docs.langchain.com/oss/python/langchain/middleware#summarization
I looked through the code for it, and it seems like it drops the messages after summarizing them.
Perhaps the original messages are still stored elsewhere in the state though, as you would need them for display in a chat based interface.
For Langchain, they have a concept of a checkpointer that can store state, so you would use a database-backed checkpointer for production, to enable restoration of the thread.

[Eoghanmcc1000]

Hi Pamela, thanks for the series. I have been watching to learn and also to prepare for upcoming interviews where I need to design and start prototyping an AI powered system in Python with LLMs, with AI tools allowed. When starting from scratch in an interview setting, how would you go about it? Do you have a particular stack you rely on, and how do you frame key trade offs such as cost, latency, and scalability?

Any brief tips for live prompting and iteration would be great too: framing the brief, setting constraints, quick checks to verify outputs, deciding what to accept or discard, and when to stop iterating. Thank you.

[pamelafox]

For Python stack:

• FastAPI or Quart, so I get concurrency. Relevant blog posts:
  https://techcommunity.microsoft.com/blog/azuredevcommunityblog/why-your-llm-powered-app-needs-concurrency/4459584
  https://blog.pamelafox.org/2024/07/should-you-use-quart-or-fastapi-for-ai.html
• If I need a database connection, asyncpg + SQLAlchemy. Relevant talk:
  https://www.youtube.com/watch?v=wJ9ieWxA7lI
  https://1drv.ms/p/c/b79cd1db3ada14c5/EbI2tmidXqlDvj7lnUKduVQBlTNsK4ERSIwydXTatRrt9g?e=Nib6bc
• For connecting to AI models, I start with the lower level OpenAI package, since most of my apps use Azure OpenAI GPT models. You could use wrapper packages like litellm, anyllm, llm, etc, but then you'll often lose some of the specific features of the provider, and it's not worth it to me. Lots of developers do that for better portability across providers however.
• If I need an agent-framework for tool calling, then I'd use agent-framework, langchain, pydantic-ai, or openai-agents. They all offer similar functionality. You need to confirm they work with your target model provider and model functionality however.
• For deployment, I use Dockerfile's with Azure Container Apps, since that can be tested locally and can be portable across clouds.

I'm not sure what you have to tackle in an interview setting, but hopefully that's helpful.

[wigging]

Does the Microsoft Agent Framework support creating agents that are not LLM agents? Can I create an agent that is for controlling a robot, an agent for interacting with a database, an agent that performs some computational task, etc.? I like the idea of an agent framework to orchestrate workflows but not everything needs to use an LLM so I'm curious if such a framework can be used without an LLM.

[AndrewMichael2020]

Yes, it's orchestration-first: you can build agents that call code, robots, databases, or services without an LLM. However, features like natural-language planning or RAG obviously require an LLM; skip those and use deterministic policies/state machines. Treat the LLM as optional middleware, swap in tool-only behaviors where language isn’t needed.

[wigging]

Good to know. Are there any examples of using the agent framework like this? All of the examples that I have seen use an LLM agent.

[pamelafox]

There are a number of examples in the workflows folder, and not all of them use LLMs:
https://github.com/microsoft/agent-framework/tree/main/python/samples/getting_started/workflows
Also check out the devui folder - I know that the spam workflow there does not use LLMs:
https://github.com/microsoft/agent-framework/tree/main/python/samples/getting_started/devui

[ogbozoyan]

Thanks for shahirng