PictoNet: A Generative Pictographic System

**Notice: this initiative is haltet in favour of pictos.net living at https://github.com/hspencer/pictos-net **

PictoNet is an open-source, public good initiative by MediaFranca dedicated to building a generative language model for Augmentative and Alternative Communication (AAC). Our mission is to create a transparent, editable, and community-governed infrastructure that empowers individuals with complex communication needs (CCN) to express themselves more fully.

We are not just building a tool; we are seeding the conditions for a community-governed commons that can grow and evolve with the people it serves. This project reframes generative systems as design materials that communities can shape to reflect their own communicative priorities.

The PictoNet Ecosystem

The PictoNet project is a modular ecosystem organized across several repositories. This structure separates concerns and clarifies how different parts of the project relate to one another.

• mediafranca/pictonet (This Repository): The core generative engine. This repository contains the trained models, the source code for the 5-stage generative pipeline, the PICTOS training dataset, and foundational documentation, including the detailed technical plan.
• mediafranca/nlu-schema: The Natural Language Understanding front-end. This repository defines the semantic schema and provides the multilayered NLU pipeline (Frame Semantics, NSM, and Speech Act Classification) that transforms natural language input into structured semantic representations. Integrated as a submodule in the nlu_frontend/ directory.
• mediafranca/pictoforge: The web-based interface for round-trip authoring, interaction, and feedback (RLHF). PictoForge is the primary gateway for users, practitioners, and researchers to engage with the generative model.
• mediafranca/vcsci: The Visual Communicability and Semantic Correspondence Index. This repository stores the benchmark phrase sets and evaluation tools used to assess the clarity, effectiveness, and semantic accuracy of the generated pictograms.

About This Repository (mediafranca/pictonet)

This repository contains the core of the PictoNet project: the generative engine that translates communicative intents into accessible, semantically-structured SVG pictograms. It serves as the central hub for the technical plan, model development, data management, and scripting related to the generative pipeline.

Key Technical Concepts & Architecture Overview

PictoNet's core innovation lies in its ability to go beyond literal text-to-image synthesis, addressing the more complex task of pragmatically aware intermodal translation. This is achieved through a sophisticated 5-stage generative pipeline:

1. Semantic Analysis Frontend (nlu-schema): Deconstructs natural language input into a rich, structured semantic representation using Frame Semantics, Natural Semantic Metalanguage (NSM), and Speech Act Classification. This module outputs a hierarchical JSON object that serves as the "semantic brief" for the generation engine.
2. Conceptual Mapping Engine: Translates the semantic representation into a set of visual concepts based on a target pictogram system (e.g., ARASAAC).
3. Hybrid SVG Generation Core: Generates the final SVG using a two-stage process: an LLM creates a semantically structured SVG "scaffold," which is then refined by a diffusion model for visual aesthetics.
4. Accessibility Post-processor: Programmatically injects accessibility metadata (e.g., <title>, <desc>, role="img", aria-labelledby) into the SVG to ensure WCAG/ARIA compliance.
5. Refinement and Iteration (RLHF): Utilizes Reinforcement Learning from Human Feedback (via PictoForge) to continuously improve the model's outputs.

For a comprehensive understanding of the linguistic and cognitive foundations, architectural paradigms, data/training strategies, evaluation protocols, and accessibility engineering, please refer to the full Technical Plan.

Getting Started

This repository is primarily focused on the development and documentation of the PictoNet generative model.

Prerequisites

• Node.js (for the API layer)
• npm (for managing Node.js dependencies)
• Python (for the ML Core)
• pip (for managing Python dependencies)

Installation

1. Clone the repository with submodules:

   git clone --recurse-submodules https://github.com/mediafranca/pictonet.git
   cd pictonet

   Or if you've already cloned the repository:

   git submodule update --init --recursive

2. Install Node.js dependencies:

   npm install

3. Install Python dependencies:

   pip install -r requirements.txt

4. Install client dependencies:

   cd client && npm install && cd ..

Running the Development Environment

• Start the API Server (Node.js/TypeScript):

  npm run dev:server

• Start the Client (Web Interface):

  npm run dev:client

• Run both concurrently:

  npm run dev

ML Core Scripts

• Train the ML Model:

  npm run ml:train

• Run ML Inference:

  npm run ml:infer

Contributing

We welcome contributions from developers, designers, researchers, speech therapists, educators, and individuals with lived experience. Your insights are critical to ensuring that PictoNet remains responsive, relevant, and truly inclusive.

Please read our CONTRIBUTING.md file for detailed information on how to get involved, including our contribution workflow, communication channels, and development setup. We also highly recommend reviewing the Technical Plan to understand the project's foundational concepts and architecture.

Governance

The project operates under a community-managed, reciprocity-based governance model. We value transparent decision-making and aim to create a structure where dedicated contributors can take on greater responsibility over time. Key communication channels include:

• GitHub Issues: For bug tracking, feature requests, and managing specific tasks.
• GitHub Discussions: For broader collaboration, brainstorming, and community conversations.

License

This project is licensed under the terms of the MIT License, unless otherwise stated.