Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/actions/spelling/expect.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ diracsgm
docsy
FedCloud
fedcloud
FedLLM
fedllm
findstr
flox
galdyn
Expand Down
20 changes: 20 additions & 0 deletions content/en/users/ai/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
title: Artificial Intelligence
linkTitle: Artificial Intelligence
type: docs
weight: 80
description: Artificial Intelligence (AI) services in the EGI Infrastructure
---

## Overview

AI is becoming an important component in supporting data-driven science
across disciplines. In this context, EGI is evolving its service offering
towards an **AI-ready compute–data continuum**, where computing, data, and AI
capabilities are integrated and accessible across federated providers.

The following AI service is currently available:

- [FedLLM](./fedllm/) providing access to LLM inference capabilities via a web interface and an OpenAI-compatible API

Its dedicated section offers a detailed description of both access methods.
107 changes: 107 additions & 0 deletions content/en/users/ai/fedllm/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
---
title: FedLLM
linkTitle: FedLLM
type: docs
weight: 10
aliases:
- /users/fedllm
description: Federated LLM inference service
---

## Overview

FedLLM is an EGI service that provides access to Large Language Models (LLMs) through:

- A web-based chat interface: [chat.ai.egi.eu](https://chat.ai.egi.eu/)
- An OpenAI-compatible API: [llm.ai.egi.eu](https://llm.ai.egi.eu/)

The service enables users to generate and analyse text,
run retrieval-augmented generation (RAG) workflows,
and integrate AI capabilities into scientific applications.

## Access and Authentication

Access is managed through EGI Check-in:

1. Create an [EGI account](../../aai/check-in/signup)
2. Enrol in a supported Virtual Organisation (VO) such as
[vo.fedllm.egi.eu](https://aai-demo.egi.eu/auth/realms/id/account/#/enroll?groupPath=/vo.fedllm.egi.eu)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not using vo.access.egi.eu VO?

IMHO, vo.access.egi.eu VO should be the one used to grant access to the "free-tier" services.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me. I’ll update the documentation once the configuration is in place.

3. Access the service via [chat.ai.egi.eu](https://chat.ai.egi.eu/)
4. To use the API, generate an API key from the web interface (Account settings → Create a new secret key)
5. Use the API key to access the [llm.ai.egi.eu](https://llm.ai.egi.eu/) API endpoints

## Using AI Models

### Web Interface

After logging into the web interface:

- Select a model from the dropdown menu
- Enter your prompt in the input field
- Submit the request to receive a response

![FedLLM Web Interface](./fedllm_ui.png)

### API Usage

You can retrieve a list of available models:

```shell
curl https://llm.ai.egi.eu/v1/models \
-H "Authorization: Bearer $API_KEY" | jq .
```

You can then send a request using one of the available models:

```shell
curl https://llm.ai.egi.eu/v1/chat/completions \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-oss-120b",
"messages": [
{ "role": "user", "content": "Hello!" }
]
}' | jq .
```

For detailed API specifications, refer to the
[API reference](https://developers.openai.com/api/reference/overview).
Note that not all endpoints are supported.

For client access and development, refer to the [LiteLLM SDK](https://docs.litellm.ai/docs/#litellm-python-sdk)
or the [OpenAI Client Libraries](https://developers.openai.com/api/docs/libraries/).

<!--
## Service options

### LLMs for Researchers

- Shared service access
- Standard quotas (requests per minute, tokens per minute)

### LLMs for Communities

- Dedicated deployments
- Custom models and quotas
- Integration with community services

### On-premises LLMs

- Deployment on community-owned infrastructure
- Full control over models and data
-->
## High-level Service architecture

The FedLLM service is composed of the following components:

- **Web interface** (Open WebUI): user interaction and API key management
- **API gateway** (LiteLLM): OpenAI-compatible interface and routing
- **Inference backend** (e.g. vLLM): execution of LLM workloads on GPU resources

User data is processed only for service delivery and is not retained or reused for training.

## Usage considerations

- Available models may vary depending on your VO
- Usage may be subject to quotas and rate limits
Binary file added content/en/users/ai/fedllm/fedllm_ui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading