Asap 231 more documentation clarity

README.md

@@ -1,23 +1,58 @@
 # ASAP PDF
 
 A Rails application for navigating PDF accessibility audits. We use traditional NLP and LLM processes to prioritize and
-stratify documents, guiding stakeholders through corrective action decision-making. In the future we hope to build in
-more accessibility auditing and remediation. For additional documentation, see the [docs](./docs) folder.
+stratify documents, guiding stakeholders through corrective action decision-making. For additional documentation, see the [docs](./docs) folder.
 
+Features:
+- Scrape websites for PDF documents and harvest metadata
+- Use NLP to classify documents by category
+- Dashboards and audit workflow to help navigate decision-making
+- LLM-powered tools to summarize and perform policy analysis on documents
+
+The feature set is an approach to organize your audit for ADA compliance - the features do not remediate documents or identify which parts of documents are inaccessible.
+
+## Contact
 Fill out [this Google Form](
 https://docs.google.com/forms/d/e/1FAIpQLSf2C4uKOgCTf-nrBM7bBWRSyNDELhE6c6EaHMN5Or71vyd7fw/viewform) to connect with Code for America and learn more.
 
-## Prerequisites
+## Architecture
+ASAP has two major architectural realms, a Ruby on Rails powered user interface and [Python components](docs/python_components.md), which provide many of the document processing and AI features. The Rails-based UI is not required. The Python components may each be used as standalone services.
+
+### Ruby on Rails Application (Audit UI)
+- **Frontend**: Hotwired (Turbo + Stimulus) with Tailwind CSS
+- **Backend**: Ruby on Rails 8.0
+- **Database**: PostgreSQL
+- **Testing**: RSpec and Capybara
+
+### Python Components
+
+The application includes several Python components for PDF processing:
+
+- **Site Crawler**: Downloads PDF files and metadata from government websites
+- **Document Classifier**: Determines document types using machine learning
+- **Document Inference**: Generates LLM summaries and performs exception checks
+- **Evaluation**: Automated LLM evaluation suite
+
+For detailed information about the Python components, see the [Python Components documentation](docs/python_components.md). Architectural diagrams are available in the [Architecture documentation](docs/architecture.md).
+
+## Getting Started
+The following instructions are intended for local development on macOS or Linux environments. To run the app locally on Windows follow [the Windows setup guide](docs/windows_localdev.md) instead. The [staging and production documentation](docs/deployment.md) details running the app on Amazon Web Services (AWS), however any cloud hosting platform should be possible.
+
+### Prerequisites
 
 Before you begin, ensure you have the following installed:
 
-* Ruby 3.2.2 (we recommend using a version manager like `rbenv` or `rvm`)
-* Node.js 18.17.0 (we recommend using `nvm` for version management)
+* Ruby 3.3.4 (we recommend using a version manager like `rbenv` or `rvm`)
+* Node.js 24.4.1 (we recommend using `nvm` for version management)
 * Yarn (latest version)
-* PostgreSQL locally or in a container.
+* PostgreSQL locally or in a container
 * Docker and Docker Compose (for LocalStack AWS services in development)
+* Optional, to use the LLM features, API credentials for Google, Anthropic or OpenAI
+
+### Running the App
+To run all the App's features, you will need to run the Rails application and Python components in separate terminal processes. For simplicity’s sake, we run the Rails application locally (on the host machine), while the Python components run in a Docker Compose (see [docker-compose.yml](docker-compose.yml)). For the LLM features to function correctly, credentials must be added to the Rail UI while the Python components are running. This simulates the production environment most directly. 
 
-## Rails App Development Setup
+#### Set up the Rails App 
 
 1. Clone the repository:
    ```bash
@@ -35,97 +70,47 @@ Before you begin, ensure you have the following installed:
    yarn install
    ```
 
-4. Setup the database:
+4. Set up the database:
    ```bash
-   bin/rails db:setup
+    rails db:migrate ; rails db:setup
    ```
 
-## Running the Application
-
-Start the development server and all required processes:
-
-```bash
-bin/dev
-```
-
-This command starts the following processes (defined in `Procfile.dev`):
-
-- Rails server
-- JavaScript build process (with esbuild)
-- CSS build process (with Tailwind CSS)
-
-The application will be available at http://localhost:3000
-
-## Rails Architecture Overview
-
-- **Frontend**: Built with Hotwired (Turbo + Stimulus) and Tailwind CSS
-- **Backend**: Ruby on Rails 7.0
-- **Testing**: RSpec, Capybara
-
-## Python Components
-
-The application includes several Python components for PDF processing:
-
-- Site Crawler: Downloads PDF files and their metadata from government websites
-- Document Classifier: Determines document types using ML
-- Document Inference: LLM summary and exception check
-- Evaluation: Automated LLM evaluation suite.
-
-To set up the Python components, follow [these instructions](python_components/README.md).
-
-**Required: Also see "Setting API credentials" below.**
-
-## Testing
-
-Run the test suite:
-
-```bash
-bundle exec rails test:prepare
-bundle exec rspec
-```
+5. Test running the app:
 
-## Development Tools
-
-The project includes several development tools:
-
-- **Brakeman**: Security analysis (`bin/brakeman`)
-- **RuboCop**: Code style checking (`bin/rubocop`)
-- **Overcommit**: Git hooks management
-- **Better Errors**: Enhanced error pages in development
-- **Bullet**: N+1 query detection
-
-## API
-
-### Setting API credentials
-
-For the Python components to use the Rails API, an API username and password must be provided in AWS secrets manager for production or in the Rails credential file for local development.
-
-To set credentials for local development add an `api_user` and `api_password` configuration via editing the credentials file:
-
-`EDITOR="Your editor" rails credentials:edit --environment development`
-
-After setting the API credentials, navigate to the AI configuration screen (http://localhost:3000/configuration/edit) and save. This will set the secrets value in the LocalStack secretsmanager service.
-
-### Sites API (v1)
-
-Some basic API endpoints are currently provided.
-
-- `GET /api/v1/sites`
-    - Lists all sites
-    - Returns site details excluding user_id, created_at, updated_at
-    - Includes s3_endpoint for each site
-
-- `GET /api/v1/sites/:id`
-    - Retrieves a specific site
-    - Returns site details excluding user_id, created_at, updated_at
-    - Includes s3_endpoint
+   ```bash
+   bin/dev
+   ```
 
-### Document Inference (v1)
+6. A default admin user was created during database setup. See [seeds.rb](db/seeds.rb) for details. To add your own admin user, run the following rake task:
+   ```bash
+   rake users:create_admin"[<your email>,<your password>]"
+   ```
+
+7. Add admin credentials for API usage by Python components. For local development, add admin credentials to the development configuration as `api_user` and `api_password`.
+   ```bash
+   `EDITOR="Your editor" rails credentials:edit --environment development`
+   ```
+   The final configuration should look something like:
+   ```yaml
+   api_user: <your email>
+   api_password: <your password>
+   ```
 
-- `POST /api/v1/documents/inference`
-    - Adds or updates a document inference record.
-    - Ideally used for storing AI results for documents.
+#### Set up the Python Components
 
+1. From the project root, build images
+   ```bash
+   docker compose build
+   ```
+2. Start the containers
+   ```bash
+   docker compose up
+   ```
+3. In a separate terminal process, run the Rails app.
+   ```bash
+   bin/dev
+   ```
+4. Log into the app, visit `/configuration/edit`. Enter just LLM API credentials you'd like to use and save the form. The credentials are ephemerally stored in the LocalStack clone of AWS SecretsManager. If you restart the Docker containers, the API credentials must be reentered.
 
 ## Adding Sites, Documents and Users
 
@@ -140,18 +125,11 @@ When the database is set up (`bin/rails db:setup`), it is populated with some sa
 
 ### Adding Documents
 - Documents may be imported via the document import rake task: `bin/rake documents:import_documents"[<site id>, <path to csv>, <archive bool*>]"`
-- Sample csv documents may be found in db/seeds. The format should match the output of the crawl and classification processes. Check out the documentation in [the python_components directory](python_components/README.md) for more details.
+- Sample csv documents may be found in db/seeds. The format should match the output of the crawl and classification processes. Check out the documentation in [the python_components directory](docs/python_components.md) for more details.
 
 *Set to true to import a csv inside a zip archive.
 
-Users, Sites and Documents may be added manually via the Rails console as well. 
-
-## Contributing
-
-1. If contributing in Ruby, ensure all tests pass and no new RuboCop violations are introduced. If contributing in
-   Python, ensure all tests pass and that no new python linting violations are introduced.
-2. Update documentation as needed
-3. Follow the existing code style and conventions
+Users, Sites and Documents may be added manually via the Rails console as well.
 
 ## License
 

app/controllers/configurations_controller.rb

@@ -5,26 +5,41 @@ class ConfigurationsController < AuthenticatedController
 
   def initialize
     super
+    @config = {
+      google_ai_api_key: nil,
+      anthropic_api_key: nil,
+      openai_api: nil,
+      google_evaluation_service_account_credentials: nil,
+      google_evaluation_sheet_id: nil,
+      localstack_not_reachable: false
+    }
+
     @secret_manager = AwsLocalSecretManager.new
-    @secret_names = Rails.configuration.local_secret_names
+
+    if Rails.configuration.respond_to?(:local_secret_names)
+      @secret_names = Rails.configuration.local_secret_names
+    else
+      @config["localstack_not_reachable"] = true
+    end
   end
 
   def edit
-    @config = {
-      localstack_not_reachable: false
-    }
-    response = @secret_manager.get_secret!(@secret_names[:google_api])
-    @config["google_ai_api_key"] = response.secret_string if response.present?
-    response = @secret_manager.get_secret!(@secret_names[:anthropic_api])
-    @config["anthropic_api_key"] = response.secret_string if response.present?
-    response = @secret_manager.get_secret!(@secret_names[:openai_api])
-    @config["openai_api"] = response.secret_string if response.present?
-    response = @secret_manager.get_secret!(@secret_names[:google_eval_service_account])
-    @config["google_evaluation_service_account_credentials"] = response.secret_string if response.present?
-    response = @secret_manager.get_secret!(@secret_names[:google_eval_sheet_id])
-    @config["google_evaluation_sheet_id"] = response.secret_string if response.present?
-  rescue Seahorse::Client::NetworkingError
-    @config["localstack_not_reachable"] = true
+    unless @config["localstack_not_reachable"]
+      begin
+        response = @secret_manager.get_secret!(@secret_names[:google_api])
+        @config["google_ai_api_key"] = response.secret_string if response.present?
+        response = @secret_manager.get_secret!(@secret_names[:anthropic_api])
+        @config["anthropic_api_key"] = response.secret_string if response.present?
+        response = @secret_manager.get_secret!(@secret_names[:openai_api])
+        @config["openai_api"] = response.secret_string if response.present?
+        response = @secret_manager.get_secret!(@secret_names[:google_eval_service_account])
+        @config["google_evaluation_service_account_credentials"] = response.secret_string if response.present?
+        response = @secret_manager.get_secret!(@secret_names[:google_eval_sheet_id])
+        @config["google_evaluation_sheet_id"] = response.secret_string if response.present?
+      rescue Seahorse::Client::NetworkingError
+        @config["localstack_not_reachable"] = true
+      end
+    end
   end
 
   def update

app/views/configurations/edit.html.erb

@@ -5,7 +5,7 @@
         <i class="fas fa-wand-magic-sparkles mr-2"></i>
         AI Configuration Settings
       </h2>
-      <% if Rails.env == "production" %>
+      <% if Rails.env == "production" || Rails.env == "staging" %>
         <p>The configuration items provided on this page are not for the production environment. Please set secrets via
           configuration or directly in AWS.</p>
       <% elsif @config["localstack_not_reachable"] == true %>
@@ -80,22 +80,26 @@
                 </div>
               </div>
             </div>
-            <div class="form-control w-full">
-              <%= f.label :google_evaluation_service_account_credentials, "Google Credentials JSON (Evaluation)", class: "label" %>
-              <div class="mt-2">
-                <%= f.text_area :google_evaluation_service_account_credentials,
-                                value: @config["google_evaluation_service_account_credentials"],
-                                class: "input input-bordered w-full font-mono" %>
+            <fieldset class="fieldset bg-base-200 border-base-300 rounded-box w-xs border p-4">
+              <legend class="fieldset-legend font-semibold">Evaluation Suite</legend>
+              <p class="text-sm mb-2">The following optional settings are only required for running the LLM evaluation suite locally.</p>
+              <div class="form-control w-full">
+                <%= f.label :google_evaluation_service_account_credentials, "Google Credentials JSON", class: "label" %>
+                <div class="mt-2">
+                  <%= f.text_area :google_evaluation_service_account_credentials,
+                                  value: @config["google_evaluation_service_account_credentials"],
+                                  class: "input input-bordered w-full font-mono py-2" %>
+                </div>
               </div>
-            </div>
-            <div class="form-control w-full">
-              <%= f.label :google_evaluation_sheet_id, "Google Sheet ID (Evaluation)", class: "label" %>
-              <div class="mt-2">
-                <%= f.text_field :google_evaluation_sheet_id,
-                                 value: @config["google_evaluation_sheet_id"],
-                                 class: "input input-bordered w-full font-mono" %>
+              <div class="form-control w-full">
+                <%= f.label :google_evaluation_sheet_id, "Google Sheet ID", class: "label" %>
+                <div class="mt-2">
+                  <%= f.text_field :google_evaluation_sheet_id,
+                                   value: @config["google_evaluation_sheet_id"],
+                                   class: "input input-bordered w-full font-mono" %>
+                </div>
               </div>
-            </div>
+            </fieldset>
             <div class="card-actions justify-end mt-6">
               <%= f.submit "Save Changes",
                            class: "btn btn-primary" %>

docs/README.md

@@ -1,7 +1,7 @@
 # Documentation
 * [Architecture](architecture.md)
 * [Deployment](deployment.md)
-* [Python Components](../python_components/README.md)
+* [Python Components](python_components.md)
 * [Terraform/OpenTofu](../terraform/README.md)
 * [Todo - Loose development roadmap](todo.md)
 * [Local Development on Windows](windows_localdev.md)

docs/deployment.md

@@ -4,8 +4,7 @@ This app deploys to AWS using automated GitHub Actions. Choose either staging or
 
 ## What You Need
 - AWS account
-- Domain name (optional but recommended for security)
-- Fork this repository
+- Domain name 
 
 ## Setup Steps
 

python_components/README.md → docs/python_components.md

@@ -8,7 +8,7 @@ To set API keys for AI services, visit the application configuration page. API k
 
 ## Prerequisites
 - Docker and Docker Compose
-- Google Cloud account with access to Gemini API or Anthropic account with API access.
+- Optional, to use the LLM features, API credentials for Google, Anthropic or OpenAI
 
 ## Code style
 
@@ -26,7 +26,7 @@ After running these scripts, the output from the classification component can be
 
 ### Helper Script
 
-A shell script has been included that combines the crawling and classification steps. You can find it in [bin/crawl](../bin/crawl). Before running the script add entries to [python_components/crawler/config.json](crawler/config.json).
+A shell script has been included that combines the crawling and classification steps. You can find it in [bin/crawl](../bin/crawl). Before running the script add entries to [../python_components/crawler/config.json](crawler/config.json).
 
 Sample usage: