# Data flow

How a flow execution travels through the system, end-to-end.

## Flow execution sequence

```mermaid
sequenceDiagram
    participant User
    participant UI as UI
    participant API as FastAPI
    participant Exec as Flow executor
    participant PB as PocketBase
    participant LLM as LLM gateway
    participant LF as Langfuse (optional)

    User->>UI: Configure flow + click Run
    UI->>API: POST /flows/execute
    API->>Exec: Start execution_id
    API-->>UI: { execution_id }
    Note over Exec: Topological sort over edges

    Exec->>LF: Open trace (execution_id)
    Exec->>PB: load_dataset(dataset_id)
    PB-->>Exec: golden items
    Exec->>LF: Span: dataset.load

    loop for each item
        Exec->>LLM: chat/completions
        LLM-->>Exec: actual_output
        Exec->>LF: Generation
    end
    Exec->>LF: Span: ai_service.endpoint

    Exec->>Exec: DeepEval scoring (LLM-as-judge per metric)
    Exec->>LF: Span: deepeval.run + scores
    Exec->>PB: save evaluation_run + items + scores

    loop UI poll
        UI->>API: GET /flows/progress/{id}
        API-->>UI: event stream
    end
```

## RAG/MCP retrieval flow

For tool-augmented evaluation (RAG, directive lookup, agent eval):

```mermaid
sequenceDiagram
    participant Exec as Flow executor
    participant MCP as MCP client
    participant Server as MCP server<br/>(golden-retriever, etc.)
    participant LLM as LLM gateway

    Note over Exec: For each dataset item

    Exec->>MCP: tool_call(query, language, country)
    MCP->>Server: JSON-RPC tool/call
    Server-->>MCP: chunks + citations
    MCP-->>Exec: retrieval_context

    Exec->>LLM: system + context + question
    LLM-->>Exec: actual_output

    Note over Exec: DeepEval scores per metric<br/>(faithfulness uses retrieval_context)
```

## Dataset item lifecycle

```mermaid
stateDiagram-v2
    [*] --> DRAFT: create
    DRAFT --> ACTIVE: approve item
    ACTIVE --> FROZEN: dataset approved
    DRAFT --> DRAFT: edit
    ACTIVE --> DRAFT: reject item
    FROZEN --> ACTIVE: admin unfreeze
    FROZEN --> [*]: dataset frozen
```

Dataset level: `DRAFT → IN_REVIEW → APPROVED`. When the dataset is `APPROVED`, all items auto-freeze (`auto_freeze_dataset_items`). Frozen items cannot be edited — only workspace admins can unfreeze.

## Persisted records per execution

| Collection           | What is saved                              | Retention                |
| -------------------- | ------------------------------------------ | ------------------------ |
| `evaluation_runs`    | run id, dataset id, summary scores, status | Indefinite (compliance)  |
| `evaluation_metrics` | per-item / per-metric numeric scores       | Indefinite               |
| `ai_reports`         | LLM-generated narrative summary            | Indefinite               |
| `flow_executions`    | nodes, edges, progress events              | 90 days                  |
| `admin_actions`      | audit log entries                          | Indefinite (RBAC + DORA) |

## When Langfuse is enabled

Per execution: **1 trace + N spans + M generations + K scores**, where N = node count, M = LLM call count, K = metric count. All instrumentation is **zero-overhead no-ops** when `LANGFUSE_*` env vars are unset.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.edge.nyami.fr/architecture/data-flow.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.