Proctoring

Documentation

Following is the list of AI proctoring components used in this project:

Pose Detection
Hand Detection
Blur Detection
Human speech detection (VAD)
Virtual Background Detection (to be added soon)

Pose Detection

1. Overview

Purpose:
This project uses an AI-based Pose detection model to determine if a user’s face (specifically the nose and eyes) is centered within a designated area on the screen. By analyzing the relative positions of key facial landmarks, it can infer if the user is looking at the lecture (i.e., maintaining focus) or showing signs of distraction. It is also used to detect the number of people in the camera feed.

Model Types:

Pose Estimation (Landmark Detection): The system identifies and tracks specific facial landmarks (nose, left eye, right eye) in real-time.

Frameworks/Libraries:

Mediapipe Pose: Provides real-time pose landmark detection.
Camera Utils (Mediapipe): Facilitates real-time video capturing from the webcam.
React: Integrates the pose detection result into a web-based frontend interface.

2. Models

General Information:

Training Dataset:
The Mediapipe Pose model is pre-trained on a large, diverse dataset of human poses and facial landmarks. The user does not need to retrain the model; it comes out-of-the-box with robust detection capabilities.
Model Source:
The model is accessed via a CDN provided by Mediapipe. It runs locally in the browser, leveraging the user’s webcam feed as input.
Potential Pitfalls:
- Lighting Conditions: Poor lighting can reduce the reliability of landmark detection.
- Camera Quality & Positioning: Low-resolution or angled webcams might fail to accurately identify landmarks.
- User Movement: Rapid head movement or partial occlusion of the face can cause intermittent detection failures.

3. Implementation

Integration (How the AI model integrates with the frontend):

Video Input Acquisition:
The code uses navigator.mediaDevices.getUserMedia({ video: true }) under the hood (through Camera from Mediapipe) to capture frames from the user’s webcam.
Pose Detection with Mediapipe:
Each frame from the webcam is processed by the Mediapipe Pose model, returning facial landmarks like nose ($(x_N, y_N)$) and eyes ($(x_{L}, y_{L}), (x_{R}, y_{R})$).
Mathematical Computations for Focus Detection:
After obtaining landmarks, the code performs geometric calculations to determine user focus:
- Checking if the Nose is Inside a Defined Box:
  A rectangular “center box” is drawn at coordinates $(x_{box}, y_{box})$ with dimensions width $W$ and height $H$. If the nose position $(x_N, y_N)$ (scaled to the canvas coordinate system) satisfies:
  $x_{box} < x_N < x_{box}+W$ and $y_{box} < y_N < y_{box}+H$
  then the user is considered “inside the frame.”
- Calculating Eye Differences:
  Let $x_{L}$ and $x_{R}$ be the normalized $x$-coordinates of the left and right eye landmarks respectively. The horizontal eye difference is:
  $eyeDiff = |x_{L} - x_{R}|$.
- Interpreting the Metrics:
  If $eyeDiff < minEyeDiffForFocus$, the system concludes that the user might be looking away or not maintaining a proper viewing angle. In that case, it increments a “look away” counter. minEyeDiffForFocus is defined at the start of the component.
By setting empirical thresholds, these distances and differences convert into actionable logic for focus detection.
User Feedback and UI Updates:
- A status message is displayed to indicate if the user is “inside the frame” or "User not in the box" or “focus on the lecture!” or "User not detected"
- A “look away” count increments whenever the user seems distracted, providing a simple metric for gauging attention over time.

4. Future Prospects

Expanded Landmark Usage:
Incorporate more facial landmarks (e.g., mouth, shoulders) for even more robust engagement metrics, such as detecting if the user is talking or nodding.
Temporal Analysis with Smoothing:
Apply temporal averaging or filtering to eye difference measurements to reduce noise and improve accuracy over multiple frames.
Multi-person Scenarios:
Extend the approach to handle multiple faces simultaneously, useful in group online sessions or lectures.
Integration with Other Modalities:
Combine pose detection with speech analysis or facial expression recognition for comprehensive user engagement tracking.

Hand Detection and Gesture Recognition

1. Overview

Purpose:
This component leverages Mediapipe’s Hand Landmarker API to detect and analyze hand gestures in real-time from a webcam feed. It determines the number of hands in the feed and identifies specific gestures, such as raised fingers, thumbs-up, or thumbs-down.

Features:

Detects up to three hands in a video feed.
Analyzes hand landmarks to identify gestures and the number of raised fingers.
Displays the detected gesture and the count of hands in the UI.

Frameworks/Libraries:

React: Provides the frontend for user interaction.
Mediapipe Tasks Vision: Handles real-time hand landmark detection.

2. Models

Model Details:

Pre-trained Mediapipe Hand Landmarker:
The model detects and tracks 21 hand landmarks, including joints and fingertips.

Input:

Video frames captured from the user's webcam.

Output:

A list of hand landmarks for each detected hand, along with gesture analysis.

Limitations:

Lighting Conditions: Poor lighting can hinder accurate detection.
Complex Gestures: Non-standard hand gestures may not be interpreted correctly.

3. Implementation

Workflow:

Video Input Acquisition:
The component accesses the user's webcam using navigator.mediaDevices.getUserMedia({ video: true }) and streams the video feed to the Hand Landmarker API.
Hand Detection:
The Mediapipe Hand Landmarker processes video frames to identify hand landmarks.
Gesture Analysis:
The following checks are performed on hand landmarks to interpret gestures:
- Finger Raised Check:
  Verifies if a finger is raised by analyzing the y-coordinates of its landmarks (MCP, PIP, DIP, TIP).
  - A finger is raised if the y-coordinate decreases monotonically from MCP to TIP and if the angle of the finger exceeds a threshold. The tangent of this angle is a configurable named minAngleTanForRaised defined at the start of the component.
- Thumb Check:
  Determines if the thumb is raised, extended, or in a downward position based on its relative position and angle. There is another configurable called minAngleTanForDown for detecting thumbs down.
- Gesture Interpretation:
  Gestures are deduced by combining the number of raised fingers with the thumb’s position. Examples include:
  - "Thumbs Up": Thumb raised, other fingers down.
  - "5": All fingers extended, thumb positioned outside the palm.
UI Updates:
The component displays:
- The detected gesture.
- The number of hands in the feed.

4. Future Prospects

Additional Gesture Support:
Extend the logic to detect more complex hand gestures (e.g., "peace" or "OK" signs).
Multi-Hand Gesture Analysis:
Implement logic to analyze interactions between multiple hands in the feed.
Real-Time Feedback:
Display live visual overlays on detected hands, such as bounding boxes or labels for raised fingers.
Robustness Improvements:
Apply noise reduction techniques to improve gesture recognition in challenging environments (e.g., poor lighting, occlusions).

VAD

1. Overview

Purpose: The Voice Activity Detection (VAD) system aims to detect speech activity in real-time audio streams, helping to distinguish between speech and non-speech segments. This can be applied in making sure that the student focuses on the content and is not distracted in speaking to someone else. This will also ensure that the student is not receiving any kind of verbal help.
Model Types: Classification Model
Frameworks/Libraries:
- MediaPipe (for audio classification and hands detection)
- TensorFlow Lite (for running the model efficiently in the browser)
- JavaScript (for handling the front-end and integration with audio APIs)

2. Models

General Information:
- The primary model used for voice activity detection is a pre-trained audio classifier, utilizing a TensorFlow Lite model (yamnet.tflite). This model is capable of detecting various sound categories, including speech, based on audio input.
- The classifier is designed to run efficiently in the browser environment, making it suitable for real-time audio classification.
- Model Source: The yamnet.tflite model is pre-trained and can be obtained from TensorFlow or similar sources.
- Potential Pitfalls:
  - Audio Quality: The accuracy of the VAD system depends on the quality of the microphone input. Poor-quality microphones or noisy environments may lead to false positives or false negatives in speech detection.
  - Real-Time Processing: Handling real-time audio streams in the browser can be computationally intensive, which may lead to delays or performance issues on lower-end devices.

3. Implementation

Integration:
- The voice activity detection system is integrated with a React frontend using the MediaPipe API for audio classification.
- The frontend uses an AudioContext to capture audio from the user's microphone. The captured audio is processed in real-time, and the yamnet.tflite model is used to classify the audio data to determine whether speech is present.
- AudioClassifier from MediaPipe is initialized, and a script processor is used to handle the audio data in chunks.
- The system continuously checks for the presence of speech and logs "Speaking detected" when it is heard.

4. Future Prospects

Model Improvements: Future versions of the system could decrease computation intensity by integrating more specialized models.

Blur Detection Documentation

1. Overview

Purpose:
The blur detection component evaluates the sharpness of an image or video frame. It is useful for quality control in image processing pipelines, ensuring that captured images meet a certain clarity threshold before being processed or analyzed.

Model Types:

Image Quality Assessment (Non-ML or ML-based):
Traditional methods rely on statistical metrics of image gradients. ML-based methods can classify images as “blurry” or “sharp” using learned features.

Frameworks/Libraries:

OpenCV: Commonly used for applying gradient operators (like Laplacian) and computing variance-based blur metrics.
TensorFlow or PyTorch (Optional): If using a learned model that classifies blur, these frameworks train and run convolutional networks that discriminate between sharp and blurry images.

2. Models

General Information:
For a simple non-ML approach, blur detection often uses the variance of the Laplacian of the image. The idea is that sharp images have strong edges and hence higher variance in the Laplacian response, while blurry images have weaker edges and lower variance.

Training Data (ML-based):
If using an ML model, training involves labeled images (e.g., a dataset of sharp vs. blurry images). For non-ML, no training is required, only setting a threshold.
Mathematics (Non-ML Approach):
Consider the grayscale image $I$. The Laplacian operator $L$ approximates the second spatial derivative: $L(x,y) = \frac{\partial^2 I}{\partial x^2} + \frac{\partial^2 I}{\partial y^2}$

In discrete form, this is done using a kernel (e.g.): $\begin{bmatrix} 0 & 1 & 0 \\ 1 & -4 & 1 \\ 0 & 1 & 0 \end{bmatrix}$

After computing the Laplacian image $L$, we calculate its variance: $\sigma^2 = \frac{1}{N}\sum_{x,y}(L(x,y) - \mu)^2$ where $\mu$ is the mean of the Laplacian values and $N$ is the total number of pixels.

A higher variance indicates a sharper image. By setting a threshold $T$:
- If $\sigma^2 < T$, the image is considered blurry.
- If $\sigma^2 \geq T$, the image is considered sharp.
Potential Pitfalls:
- Lighting and Noise: Very noisy images might appear as “sharp” due to high gradient variance.
- Texture Variation: Some images are naturally low in texture (e.g., sky), leading to low variance without being truly “blurry.”
- Threshold Selection: Choosing a universal threshold $T$ is challenging due to scene variability.

3. Implementation

Integration with the Frontend:

Image Acquisition:
The frontend captures or receives an image frame from a webcam or file input.
Laplacian Computation:
Using OpenCV.js in the browser, apply the Laplacian operator to the image.
Variance Calculation:
Compute the variance of the resulting Laplacian image and compare it to a chosen threshold.
User Feedback:
Display a message or visual indicator if the image is deemed blurry. This could prompt the user to capture another image or adjust their camera focus.

4. Future Prospects

ML-based Classification:
Incorporate a CNN trained to distinguish blurred and sharp images under various conditions, improving robustness over simple variance-based methods.
Adaptive Thresholding:
Dynamically adjust the threshold based on scene content or use machine learning to determine scene-specific thresholds.
Integration with Other Quality Metrics:
Combine blur detection with other image quality metrics (e.g., noise level, exposure) for comprehensive quality assessment.

Snapshot Storage System Documentation

Overview

The Snapshot Storage System is a robust client-side mechanism to store, retrieve, and manage snapshot data, leveraging IndexedDB and JavaScript utilities. It supports capturing video frames and screenshots, maintaining a fixed memory capacity, and automatically removing old snapshots.

Architecture

Components

dbUtils.ts
Handles direct interactions with IndexedDB, such as storing, retrieving, and deleting snapshots.
snapUtils.ts
Provides high-level utility functions for managing snapshots, including frame and screenshot capture, and automatic database cleanup.

Database Design

Database Details

Database Name: ProctoringAppDB
Object Store Name: snapshots
Key Path: id (auto-incremented)

Fields

Field	Type	Description
`id`	`number`	Auto-incremented unique identifier for each snapshot.
`image`	`string`	Base64-encoded image of the captured video frame.
`screenshot`	`string`	Base64-encoded screenshot of the page (via `html2canvas`).
`anomalyType`	`string`	Description of the anomaly detected (e.g., "blur video").
`timestamp`	`string`	ISO 8601 timestamp indicating when the snapshot was recorded.

`dbUtils.ts` Functions

1. Save Snapshot

Description

Stores a new snapshot in the database.

Parameters

image (string): Base64-encoded image of the snapshot.
screenshot (string): Base64-encoded screenshot of the page.
anomalyType (string): Type of anomaly associated with the snapshot.
timestamp (string): Time of creation in ISO 8601 format.

Returns

Promise<number>: Resolves with the auto-incremented id of the saved snapshot.

2. Retrieve All Snapshots

Description

Fetches all snapshot records from the database.

Returns

Promise<Array>: Resolves with an array of all snapshot records.

3. Delete a Snapshot

Description

Deletes a specific snapshot by its id.

Parameters

id (number): Unique identifier of the snapshot to delete.

Returns

Promise<void>: Resolves when the snapshot is deleted.

4. Clear All Snapshots

Description

Clears all entries in the snapshots object store.

Returns

Promise<void>: Resolves when the object store is cleared.

`snapUtils.ts` Functions

1. Handle Save Snapshot

Description

Captures a video frame and a screenshot, then saves the data in the database. Automatically deletes the oldest snapshot if the total exceeds the memory capacity.

Parameters

anomalyType (string): Description of the detected anomaly.
video (HTMLVideoElement): Reference to the video element capturing the feed.

Returns

Promise<number | undefined>: Resolves with the ID of the saved snapshot or undefined if saving fails.

Example Usage

import { handleSaveSnapshot } from "./snapUtils";

const videoElement = document.querySelector("video");
handleSaveSnapshot({
  anomalyType: "blur video",
  video: videoElement,
});

2. Capture Frame

Description

Captures a single frame from the video feed and converts it to a Base64-encoded image.

Parameters

video (HTMLVideoElement): Reference to the video element.

Returns

Promise<string | undefined>: Base64-encoded image or undefined if capturing fails.

3. Capture Screenshot

Description

Takes a screenshot of the current page using html2canvas.

Returns

Promise<string | undefined>: Base64-encoded screenshot or undefined if capturing fails.

4. Delete Old Snapshot

Description

Deletes the oldest snapshot if the total exceeds the memory capacity.

Parameters

added_id (number): The ID of the most recently added snapshot.

Returns

Promise<void>: Resolves when the old snapshot is deleted.

Memory Management

Memory Capacity

Capacity: The database can hold up to 20 entries at a time (memoryCapacity is configurable).
Automatic Cleanup:
- When a new snapshot is saved, the system checks if the total entries exceed the capacity.
- The oldest snapshot is deleted to maintain the capacity.

Testing and Validation

Test Cases

Save and Retrieve Snapshots
Capture video frames and screenshots, then verify their retrieval.
Auto-Increment ID Validation
Ensure IDs are correctly incremented.
Memory Capacity Enforcement
Change the memoryCapacity configurable and ensure that the change reflects in the length of the retrieved snapshots array.
Error Handling
Simulate errors (e.g., missing video feed) and ensure proper error messages are logged.

Dependencies

Libraries

html2canvas
Captures screenshots of the current page.
GitHub Repository

Future Enhancements

Filtered Retrieval
Add support for retrieving snapshots based on criteria (e.g., timestamp or anomaly type).
Remote Sync
Implement backup and synchronization with a server for persistent storage.
Optimized Storage
Compress Base64 images to save storage space.

SnapshotRecorder Component Documentation

Overview

The SnapshotRecorder component is a React functional component designed to capture snapshots of a video feed at regular intervals. It monitors anomalies in user behavior, maintains a counter for anomalies, and uploads a sequence of frames to the cloud if an anomaly is detected. The system ensures efficient storage and upload management by adhering to defined capacity limits.

Component Details

Props

`anomalies`

Type: Anomalies (Interface)
Description: An object containing live anomaly data used to determine if a snapshot should be recorded and uploaded.

Fields:

Field	Type	Description
`lookAwayCount`	`number`	Tracks instances where the user is looking away.
`numPeople`	`number`	The number of people detected in the video feed.
`handCount`	`number`	The number of hands detected in the frame.
`isBlur`	`string`	Indicates if the video feed is blurred (`"Yes"` or `"No"`).
`status`	`string`	Status of the user (e.g., `"User not detected."`).

Key Features

1. Video Feed Initialization

The component initializes the user's webcam on mount and assigns the video stream to a videoRef.
Function: startWebcam
Error Handling: Stops all video tracks on unmount to release resources.

2. Snapshot Recording

Snapshots are captured every 2 seconds using the handleSaveSnapshot function.
Anomalies are evaluated via the getActiveAnomalies helper function.

3. Counter-Based Frame Upload

The idea is to upload counterLimit number of snapshots both before and after the detection of an anomaly to cloud for persistent storage.

If an anomaly is detected:
1. A counter is started which is decremented after saving each snapshot.
2. While the counter is active, snapshots are stored without resetting the counter.
3. When the counter reaches 1, the last 2 * counterLimit + 1 (2 * counterLimit on either side of the frame with the anomaly, + the frame containing the anomaly) frames are uploaded to the cloud using the upload function.

Note: It is crucial that the number of snapshots uploaded to cloud at any instance does not exceed the memory capacity defined in the snapUtils.ts file. ie.,

memoryCapacity >= 2 * counterLimit + 1

4. Anomaly Detection

Anomalies are evaluated using getActiveAnomalies, which checks:
- User looking away for extended periods.
- Multiple people detected in the frame.
- More than 2 hands visible.
- Blurred video feed.
- Specific user statuses (e.g., "User not detected.").

Helper Functions

`getActiveAnomalies`

Description: Evaluates the current anomalies prop to determine active anomalies.
Returns: A comma-separated string of active anomalies or an empty string if none are detected.

Active Anomalies:

"looking away": When lookAwayCount is a positive multiple of 5000.
"multiple people": When numPeople > 1.
"more than two hands": When handCount > 2.
"blur video": When isBlur === "Yes".
Specific status messages (e.g., "User not detected.").

Component Lifecycle

`useEffect` 1: Webcam Initialization

Runs on mount to start the webcam feed.
Calls clearSnapshots to ensure the database is empty before capturing new snapshots.
Stops video tracks on unmount.

`useEffect` 2: Snapshot Recording

Sets up an interval to capture snapshots every 2 seconds.
Handles anomaly detection and counter-based uploads:
- Captures snapshots using handleSaveSnapshot.
- Uploads the last 2 * counterLimit + 1 frames when the counter reaches 1.

`useEffect` 3: Prop Updates

Updates the currentAnomaliesRef whenever the anomalies prop changes.

Constants

`counterLimit`

Type: number
Default: 5
Description: Controls the counter for anomaly-based uploads. The total number of frames uploaded is 2 * counterLimit + 1.

Functions

1. `startWebcam`

Description: Initializes the webcam feed and assigns the video stream to the videoRef.
Error Handling: Ensures video tracks are stopped on unmount.

2. `handleSaveSnapshot`

Import: From ../../lib/snapUtils.ts
Description: Captures the current video frame, evaluates anomalies, and stores the snapshot in IndexedDB.
Returns: The id of the saved snapshot or undefined on failure.

3. `upload`

Import: From ../../lib/cloudUtils.ts
Description: Uploads the last 2 * counterLimit + 1 frames to the cloud when anomalies are detected.

Dependencies

Libraries

React: Provides the functional component framework.
html2canvas: Used indirectly via snapUtils.ts for capturing screenshots.
IndexedDB: Used for local storage of snapshots.
Custom Modules:
- snapUtils.ts: For managing snapshot capture and storage.
- dbUtils.ts: For clearing snapshots from IndexedDB.
- cloudUtils.ts: For uploading frames to the cloud.

Usage Example

import SnapshotRecorder from "./SnapshotRecorder";

const anomalies = {
  lookAwayCount: 5000,
  numPeople: 2,
  handCount: 3,
  isBlur: "Yes",
  status: "User not detected.",
};

function App() {
  return <SnapshotRecorder anomalies={anomalies} />;
}

export default App;

Testing and Validation

Test Cases

Webcam Initialization:
- Verify that the webcam feed starts and stops correctly.
Snapshot Capture:
- Confirm snapshots are stored in IndexedDB every 2 seconds.
Anomaly Detection:
- Test scenarios with various combinations of anomalies.
Counter-Based Uploads:
- Trigger uploads by simulating anomalies and verify the correct number of frames are uploaded.
Error Handling:
- Ensure the component logs meaningful errors for missing video feed or snapshot failures.

Notes

The upload capacity (2 * counterLimit + 1) must not exceed the memory capacity defined in snapUtils.ts.
The component is designed to efficiently manage resources by clearing old snapshots and stopping unused video streams.

Future Enhancements

Dynamic Memory Management:
- Adjust memory and upload capacities based on system performance.

Dhananjaya Lab for Education Design IIT Ropar

Proctoring

Documentation

Pose Detection

1. Overview

2. Models

3. Implementation

4. Future Prospects

Hand Detection and Gesture Recognition

1. Overview

2. Models

3. Implementation

4. Future Prospects

VAD

1. Overview

2. Models

3. Implementation

4. Future Prospects

Blur Detection Documentation

1. Overview

2. Models

3. Implementation

4. Future Prospects

Snapshot Storage System Documentation

Overview

Architecture

Components

Database Design

Database Details

Fields

dbUtils.ts Functions

1. Save Snapshot

Description

Parameters

Returns

2. Retrieve All Snapshots

Description

Returns

3. Delete a Snapshot

Description

Parameters

Returns

4. Clear All Snapshots

Description

Returns

snapUtils.ts Functions

1. Handle Save Snapshot

Description

Parameters

Returns

Example Usage

2. Capture Frame

Description

Parameters

Returns

3. Capture Screenshot

Description

Returns

4. Delete Old Snapshot

Description

Parameters

Returns

Memory Management

Memory Capacity

Testing and Validation

Test Cases

Dependencies

Libraries

Future Enhancements

SnapshotRecorder Component Documentation

Overview

Component Details

Props

anomalies

Key Features

1. Video Feed Initialization

2. Snapshot Recording

3. Counter-Based Frame Upload

4. Anomaly Detection

Helper Functions

getActiveAnomalies

`dbUtils.ts` Functions

`snapUtils.ts` Functions

`anomalies`

`getActiveAnomalies`

`useEffect` 1: Webcam Initialization

`useEffect` 2: Snapshot Recording

`useEffect` 3: Prop Updates

`counterLimit`

1. `startWebcam`

2. `handleSaveSnapshot`

3. `upload`