Trusted AI - Towards a Curator (TAITaC) is an initiative dedicated to constructing metadata and curating data crucial for the development of AI and for scientific research at large. This project is birthed from the recognition that the landscape of AI and scientific research thrives on reliable data, and that constructing reliable metadata and curating such data automatically can revolutionize the way we approach research and AI model development.
TAITaC is part of the Notre Dame Trusted AI Knowledge Engineering
project, TAITAC aims to explore the architectures needed for “Curator
AI’s”. The project’s home is the nd-crane/taitac repository on GitHub.
- Automatic Metadata Generation: Construct rich metadata for AI development, capturing necessary details and relationships.
- Data Curation: Automate the process of sorting, categorizing, and maintaining data, ensuring AI models and researchers have access to the best quality data.
- Curator AI Architectures Experimentation: Experiment and iterate on potential AI architectures to best serve the goals of data curation and metadata generation.
“Moreover, the semantic interoperability layer is fundamental in most of the cases. In addition, ontologies and taxonomies combined with AI can help in establishing interoperability between different systems. The solutions analysed classify, detect and provide structure, among other actions performed on data. Hence, AI has the capability to standardise, clean, structure and increase the usage of large volumes of data, thus improving overall quality and making it easier to use and share between different systems.”
- Language: Python
- Development Framework: nbdev
- Environment Management: miniforge
- Development Environment: devcontainers
- AI Interaction: llm python library (For Large Language Model APIs interaction)
- Knowledge Graph: rdflib (For RDF graph construction and querying)
- Install Python 3.9 or higher.
- Install miniforge.
- Recommended: Visual Studio Code with devcontainers extension.
-
Clone the
nd-crane/taitacrepository:git clone https://github.com/nd-crane/taitac.git cd taitac -
Set up the devcontainer environment (if using VS Code):
- Open VS Code and open the project folder.
- When prompted, reopen the folder in the devcontainer.
-
Create a conda environment using miniforge:
conda create -n taitac_env python=3.9 conda activate taitac_env -
Install the required dependencies:
pip install -r requirements.txt
To be added as the project progresses.
We welcome contributions to the TAITAC project! If you’d like to contribute, please see our CONTRIBUTING.md for guidelines and details.
This project is licensed under the MIT License.
- Notre Dame Trusted AI Knowledge Engineering project for providing the foundational context.
- All contributors and supporters of the project.
For more details, updates, and discussions, please refer to the official GitHub repository and the associated issues and pull requests.
Our approach is a comprehensive blend of modern technologies and strategies to build a robust, interactive, and intelligent data platform. At its core, it’s an ecosystem that harmonizes Knowledge Graphs (KG) with cognitive capabilities of Large Language Models (LLM), streamlined by efficient data interfaces like SPARQL, JSON-LD, and OpenAPI.
We roughly follow the unified framework for LLMs proposed by Wang et al. (2021) to build our agents. The framework is shown below:
OpenAI has announced a Agents framework called the Assistants - OpenAI API that allows users to build agents that can interact with humans. The components of the assistants API roughly decomposes to the following Agent Architectural Patterns:
| Unified Framework Component | Assistants API Equivalent | Description |
|---|---|---|
| Profile Module | Assistant | Defines the AI’s attributes, capabilities, and personality, setting the stage for its interactions. |
| Memory Module | Thread | Manages the history of interactions, storing and truncating messages to maintain context. |
| Planning Module | Run | Processes the current state and plans the next actions, deciding how to utilize messages and tools. |
| Action Module | Message & Run Step | Executes actions, generates responses, and may call tools to produce outputs or perform tasks. |
State transitions in the Assistants API would involve changes in the “Thread” as new “Messages” are added, and as the “Assistant” performs new “Runs” and “Run Steps,” updating its current knowledge and planned actions.
In addition to these framework components, the Assistants API also provides a “Tool” component that allows the agent to perform tasks and produce outputs through OpenAI Functions and Retrieval Augmented Generation (RAG) models through document upload.
OpenAI has added python SDK examples to it’s cookbook Assistants API Overview (Python SDK) and LLamaIndex has integrated examples of using the Assistants API including ReAct Agents.
Simon Willison has a very simple blog post of implementing the ReAct: Synergizing Reasoning and Acting in Language Models in one of his TILs – A simple Python implementation of the ReAct pattern for LLMs. There are some issues in using regular expressions to select the tool the LLM want’s to use. If the response can be constrained using a grammar as with the OpenAI model, more expressive response options are available.
The issue with the ReAct pattern is that it is course grained planning and action. A new approach, called ADaPT: As-Needed Decomposition and Planning with Language Models allows recursive decomposition of the planning and action steps as needed by supporting logical operators within the task planning and execution.
Column Type Annotation using ChatGPT is a novel approach to annotate the semantic types of table columns using a large language model (LLM) called ChatGPT¹[1]. The proposed method is based on the idea that a language model can be used to generate natural language descriptions of table columns, which can then be used to infer their semantic types. The method is evaluated on the WikiTableQuestions dataset²[2] and achieves state-of-the-art performance in zero-shot and few-shot settings.
- Column Type Annotation using ChatGPT: A novel approach to annotate the semantic types of table columns using a large language model (LLM) called ChatGPT¹[1].
- Prompt Design and Evaluation: Different ways to formulate prompts for the column type annotation (CTA) task and their performance in zero- and few-shot settings.
- Explicit Instructions and Message Roles: How to improve the performance of ChatGPT by providing step-by-step instructions and using message roles to distinguish between system, user, and AI messages²[2].
- In-Context Learning: How to further boost the performance of ChatGPT by providing task demonstrations as part of the prompt in one-shot and five-shot setups.
- Two-Step Pipeline: A proposed method to deal with large label spaces by first predicting the domain of the table and then using only the relevant subset of labels for CTA³[3].
- Comparison to Baselines: A comparison of ChatGPT to state-of-the-art CTA methods based on pre-trained language models (PLMs) such as RoBERTa and DODUO.
The GitTables: A Large-Scale Corpus of Relational Tables has constructed a test data set of 1.5 million tables from GitHub. The data set is available from [GitTables]( - GitTables benchmark - column type detection
OpenAI uses the Evals Framework for unit and behavior testing LLMs as a potential model for applying trusted AI to the models. A informative blog post is at Mastering OpenAI’s ‘evals’: A Deep Dive into Evaluating LLMs and OpenAI touched on them during developer days The New Stack and Ops for AI. LlamaIndex also has a section in it’s documentation on Evaluation. LLamaIndex can use deepeval. RAG pipelines can also be evaluated with Ragas.
- Bad outputting format
- Inaccurate responses/actions
- “Going off the rails”
- Bad tone
- Hallucinations
- Model-graded evals – use GPT4 as a grader