LlamaIndex 一周年：多模态 RAG 与 OpenAI 助手集成等更新

Hello Llama Friends 🦙

LlamaIndex is 1 year old this week! 🎉 To celebrate, we’re taking a stroll down memory lane on our blog with twelve milestones from our first year. Be sure to check it out.

Explore our free and paid plans today.

Last week we had a blast with all the new things from OpenAI Dev day to learn and explore at LlamaIndex. There was a special edition newsletter with the things we released the same day as the conference, but this week’s newsletter is full of follow-up releases and explorations — don’t miss our slide deck summing up all the new features!

As always, if you’ve got a cool project or a video to share we’d love to see it! Just drop us a line at news@llamaindex.ai.

🤩 First, the highlights:

1. Multi-Modal RAG Stack: we unveiled Multi-Modal RAG ****for complex Q&A on documents and images, with new text/image queries and retrieval solutions. Notebook, Tweet, Blog post.
2. OpenAIAssistantAgent Abstractions: we released new abstractions to connect OpenAI Assistant API with any vector database. Docs, Tweet.
3. Parallel Function Calling: we enhanced our data extraction and tool execution using OpenAI’s parallel function calling. Tweet.
4. MechGPT Project: Prof. Markus J. Buehler’s work merges LLM fine-tuning with knowledge graphs for scientific discovery. Tweet, Paper.
5. Feature Slide Deck: Released a slide deck with 10+ new features and guides post-OpenAI updates.

✨ Feature Releases and Enhancements:

• We introduced a multi-modal RAG stack for complex document and image QA, featuring text/image queries, joint text/ image embeddings, and versatile storage and retrieval options. Notebook, Tweet, Blog post.
• We now offer experimental GPT-4-vision support in chat.llamaindex.ai . Users can now upload images for enhanced chatbot interactions. Tweet.
• We integrated OpenAI’s parallel function calling for efficient extraction of structured data from unstructured text and improving tool execution with agents. Tweet.
• We introduced OpenAIAssistantAgent abstractions for seamless connection of OpenAI Assistants API with your chosen vector database. Docs, Tweet.
• We introduced a new agent leveraging OpenAI Assistants API with features like in-house code interpretation, file retrieval, and function calling for external tools integration. Notebook, Tweet.

🎥 Demos:

• MechGPT by Professor Markus J. Buehler showcases the integration of LLM fine-tuning and knowledge graph creation with LlamaIndex, leading to interesting insights in cross-disciplinary scientific research and hypothesis generation. Tweet, Paper.

🗺️ Guides:

• We released a concise slide deck that aggregates over 10+ newly shipped features, guides, and analyses, complete with links to accompanying notebooks for developer use based on OpenAI’s recent updates.
• We also released a full cookbook showing how you can build advanced RAG with the Assistants API — beyond just using the in-house Retrieval tool.
• We produced a guide on evaluating the OpenAI Assistant API vs RAG with LlamaIndex.
• Here’s a guide on evaluating How well long-context LLMs (gpt-4-turbo, claude-2) recall specifics in BIG documents? (>= 250k tokens).
• Here’s another guide that highlights how function calling simplifies structured data extraction, while JSON mode ensures format correctness without schema enforcement.
• Finally, we released a guide to craft a GPT Builder, enabling an agent to programmatically construct another task-specific agent. This builder streamlines the creation of systems for specific functions. Notebook, Tweet.

✍️ Tutorials:

• Bhavesh Bhat gave us a tutorial on How to Chat with YouTube Videos Using LlamaIndex.
• David Garnitz’s tutorial blog explores the use of VectorFlow alongside ArizePhoenix, Weaviate, and LlamaIndex to manage large data sets.
• Harshad Suryawanshi’s tutorial covers Building My Own ChatGPT Vision with PaLM, KOSMOS-2 and LlamaIndex.
• Sudarshan Koirala’s made a tutorial on Creating OpenAI Assistant Agent with LlamaIndex.
• Our own Ravi Theja released his tutorial on Boosting RAG with Embeddings & Rerankers.

🎥 Webinars:

• Check out our webinar with Dan Shipper, CEO of every to talk about the implications of OpenAI’s release updates.
• A second webinar with Victoria Lin, author of the RA-DIT paper on Fine-tuning + RAG.
• Last but not least, Mayo Oshin’s webinar with Jerry Liu on How to Analyze Tables In Large Financial Reports Using GPT-4.