Introduction
Globally, over 90 million children live with visual disabilities, relying on braille books or displays that often fail to
effectively convey complex graphics or lengthy content and take considerable time to produce, limiting the learning
process. This study introduced an innovative multimodal learning assistant system to enhance tactile and textual
understanding for visually impaired students, demonstrating the potential of combining tactile and AI technologies to
improve learning outcomes , offering a significant step forward in accessible education.
Research Background & Aim
Education is a fundamental right for all, yet visually disabled (VD) students encounter distinct obstacles,
particularly in STEM disciplines where comprehending intricate graphics and diagrams is crucial. With around 90 million
visually impaired children worldwide, achieving the United Nations' Sustainable Development Goal 4 and 10—which
emphasizes inclusive and equitable quality education—is vital for their empowerment and future opportunities.
Traditional educational resources, such as Braille books and tactile graphics, frequently fall short in providing VD students
with an effective means to grasp spatial and visual concepts. To bridge this gap, personalized teaching methods and
innovative tactile learning tools play a critical role. For example, employing physical models to convey abstract concepts
like molecular structures has demonstrated significant benefits. However, these conventional solutions remain limited in
their scope and widespread accessibility.
To address these challenges, we aim to design an integrative platform that utilizes actuator arrays to convert 2D
graphics into 3D tactile perception that can enhance the spatial understanding for VD students. Additionally, integrating
augmented reality (AR) capabilities and leveraging Retrieval-Augmented Generation (RAG) systems to tailor educational
content can further enhance text comprehension and provide personalized learning experiences. Such innovations aim to
ensure that VD students receive education of the same quality and opportunity as their sighted peers, thus promoting a
more inclusive and equitable learning environment in alignment with the SDGs.
Methodology
The platform aims to provide visually impaired students with an adaptive and interactive learning experience. The
core architecture integrates a Tactile Pad for enhanced graphic perception with a Retrieval-Augmented Generation (RAG)
large language model (LLM) to improve text comprehension. Additionally, it incorporates TripoSR from VAST AI research
to transform 2D images into 3D models, enabling students to explore graphical content through dynamic feedback.
Our platform consists of 2 main components:
1. Text Comprehension Enhancement System
This system utilizes a GPT4o to retrieve relevant materials and generate customized and aligned explanations
through prompt engineering and RAG architecture, improving students' understanding of educational content.
2. Tactile Perception Enhancement System
The Tactile Pad is an interface for visualizing 3D concepts using 256 adjustable pins to create dynamic shapes. It
provides real-time tactile feedback, helping students understand complex ideas through the TripoSR algorithm that
converts 2D graphics into 3D models.
Systematic Alignment with Visually Disabled Students
Throughout development, researchers iteratively refined the system based on feedback from visually disabled students
and their teachers to meet their requirements and preferences.
Evaluation
To evaluate the system, two participants completed over 90 tasks using the VDEA Benchmark. The Experimental
Condition involved a personalized LLM and 3D tactile display, while the Control Condition used Braille books and static
tactile diagrams
An overview of how VD students utilize our system and how we evaluate the effectiveness
Development of Text Comprehension Enhacement System
To help students understand spatial and visual concepts better than conventional textbooks, we aimed to create a
personalized, visually disability focused content enhancement system through LLMs. Here, we considered 2 approaches.
1. Fine tuning a domain specific LLM for expertise
To create a LLM tailored to generate content specifically for visually disabled students. We utilized Low Rank Adapters
to fine-tune Llama3-TAIDE-LX-8B-Chat-Alpha1 with visually disabled assistant dataset. Benefits of utilizing local LLM
ensures both efficient computing and upmost privacy for visually disabled students.
2. Create a visual disability focused, universal architecture
Using advanced universal language model (e.g. GPT4o) with vector knowledge base to create a advanced learning
assistant. Benefits of it includes greater reasoning ability, which is crucial for assisting scientific and mathematic
assignment and assistance in helping understanding.