-
Notifications
You must be signed in to change notification settings - Fork 52
Add AI section with FedLLM service documentation #803
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mrorro
wants to merge
4
commits into
EGI-Federation:main
Choose a base branch
from
mrorro:fedllm-docs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,6 +17,8 @@ diracsgm | |
| docsy | ||
| FedCloud | ||
| fedcloud | ||
| FedLLM | ||
| fedllm | ||
| findstr | ||
| flox | ||
| galdyn | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| --- | ||
| title: Artificial Intelligence | ||
| linkTitle: Artificial Intelligence | ||
| type: docs | ||
| weight: 80 | ||
| description: Artificial Intelligence (AI) services in the EGI Infrastructure | ||
| --- | ||
|
|
||
| ## Overview | ||
|
|
||
| AI is becoming an important component in supporting data-driven science | ||
| across disciplines. In this context, EGI is evolving its service offering | ||
| towards an **AI-ready compute–data continuum**, where computing, data, and AI | ||
| capabilities are integrated and accessible across federated providers. | ||
|
|
||
| The following AI service is currently available: | ||
|
|
||
| - [FedLLM](./fedllm/) providing access to LLM inference capabilities via a web interface and an OpenAI-compatible API | ||
|
|
||
| Its dedicated section offers a detailed description of both access methods. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| --- | ||
| title: FedLLM | ||
| linkTitle: FedLLM | ||
| type: docs | ||
| weight: 10 | ||
| aliases: | ||
| - /users/fedllm | ||
| description: Federated LLM inference service | ||
| --- | ||
|
|
||
| ## Overview | ||
|
|
||
| FedLLM is an EGI service that provides access to Large Language Models (LLMs) through: | ||
|
|
||
| - A web-based chat interface: [chat.ai.egi.eu](https://chat.ai.egi.eu/) | ||
| - An OpenAI-compatible API: [llm.ai.egi.eu](https://llm.ai.egi.eu/) | ||
|
|
||
| The service enables users to generate and analyse text, | ||
| run retrieval-augmented generation (RAG) workflows, | ||
| and integrate AI capabilities into scientific applications. | ||
|
|
||
| ## Access and Authentication | ||
|
|
||
| Access is managed through EGI Check-in: | ||
|
|
||
| 1. Create an [EGI account](../../aai/check-in/signup) | ||
| 2. Enrol in a supported Virtual Organisation (VO) such as | ||
| [vo.fedllm.egi.eu](https://aai-demo.egi.eu/auth/realms/id/account/#/enroll?groupPath=/vo.fedllm.egi.eu) | ||
| 3. Access the service via [chat.ai.egi.eu](https://chat.ai.egi.eu/) | ||
| 4. To use the API, generate an API key from the web interface (Account settings → Create a new secret key) | ||
| 5. Use the API key to access the [llm.ai.egi.eu](https://llm.ai.egi.eu/) API endpoints | ||
|
|
||
| ## Using AI Models | ||
|
|
||
| ### Web Interface | ||
|
|
||
| After logging into the web interface: | ||
|
|
||
| - Select a model from the dropdown menu | ||
| - Enter your prompt in the input field | ||
| - Submit the request to receive a response | ||
|
|
||
|  | ||
|
|
||
| ### API Usage | ||
|
|
||
| You can retrieve a list of available models: | ||
|
|
||
| ```shell | ||
| curl https://llm.ai.egi.eu/v1/models \ | ||
| -H "Authorization: Bearer $API_KEY" | jq . | ||
| ``` | ||
|
|
||
| You can then send a request using one of the available models: | ||
|
|
||
| ```shell | ||
| curl https://llm.ai.egi.eu/v1/chat/completions \ | ||
| -H "Authorization: Bearer $API_KEY" \ | ||
| -H "Content-Type: application/json" \ | ||
| -d '{ | ||
| "model": "gpt-oss-120b", | ||
| "messages": [ | ||
| { "role": "user", "content": "Hello!" } | ||
| ] | ||
| }' | jq . | ||
| ``` | ||
|
|
||
| For detailed API specifications, refer to the | ||
| [API reference](https://developers.openai.com/api/reference/overview). | ||
| Note that not all endpoints are supported. | ||
|
|
||
| For client access and development, refer to the [LiteLLM SDK](https://docs.litellm.ai/docs/#litellm-python-sdk) | ||
| or the [OpenAI Client Libraries](https://developers.openai.com/api/docs/libraries/). | ||
|
|
||
| <!-- | ||
| ## Service options | ||
|
|
||
| ### LLMs for Researchers | ||
|
|
||
| - Shared service access | ||
| - Standard quotas (requests per minute, tokens per minute) | ||
|
|
||
| ### LLMs for Communities | ||
|
|
||
| - Dedicated deployments | ||
| - Custom models and quotas | ||
| - Integration with community services | ||
|
|
||
| ### On-premises LLMs | ||
|
|
||
| - Deployment on community-owned infrastructure | ||
| - Full control over models and data | ||
| --> | ||
| ## High-level Service architecture | ||
|
|
||
| The FedLLM service is composed of the following components: | ||
|
|
||
| - **Web interface** (Open WebUI): user interaction and API key management | ||
| - **API gateway** (LiteLLM): OpenAI-compatible interface and routing | ||
| - **Inference backend** (e.g. vLLM): execution of LLM workloads on GPU resources | ||
|
|
||
| User data is processed only for service delivery and is not retained or reused for training. | ||
|
|
||
| ## Usage considerations | ||
|
|
||
| - Available models may vary depending on your VO | ||
| - Usage may be subject to quotas and rate limits | ||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not using
vo.access.egi.euVO?IMHO,
vo.access.egi.euVO should be the one used to grant access to the "free-tier" services.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good to me. I’ll update the documentation once the configuration is in place.