Publications

App Planner: Co-creating mobile applications with Generative AI

Published in AAAI2024 Workshop on AI for Education - Bridging Innovation and Responsibility, 2023

Abstract : App Planner is an interactive tool crafted to assist students in crystallizing their ideas into actionable mobile application designs that address real-world problems. Harnessing the capabilities of generative artificial intelligence (AI), App Planner scaffolds students’ efforts towards creating mobile apps through guided conversations via a chat-based interface. Built on the principles of ‘collaborative learning’ and ‘learning by doing,’ this interface collaborates with students as a partner rather than a mentor. It assists them in brainstorming and formulating new ideas for applications, provides feedback on those ideas, and stimulates creative thinking. We mediate these conversations to follow a design thinking framework that enhances and encourages students to adopt human-centered problem-solving and critical thinking perspectives. We envision App Planner to be a catalyst for student empowerment, unlocking their potential to innovate and create with technology, guided by a human-centered AI mindset.

Download here

AI-Augmented Feature to Edit and Design Mobile Applications

Published in MobileHCI, 2023

Abstract : We are developing an AI assistance platform that enables individuals with a limited technical background, such as children, to create mobile applications from natural language input. The platform is based on MIT App Inventor and allows users to easily edit the interface and functionality of the components of their app using textual commands. The goal of the platform is to empower children and others without a background in coding with the ability to create their own mobile applications and foster their creativity and problem-solving skills in a fun and interactive way.

Download here

Digital Art Therapy with Gen AI: Mind Palette

Published in Conference of the Association for the Advancement of Affective Computing, 2023

Abstract : Proper and delightful intervention can reduce the progression of mental disorders, but numerous physical and psy- chological barriers continue to exist. In response, we developed a mobile application called “Mind Palette”, which incorporates art therapy methodologies and generative AI technology. This application aims to address the need for comprehensive inter- ventions in the mental health crisis, particularly among younger age groups. With AI chatbot interactions and AI-generated artwork recommendations, the application may facilitate dis- cussions about emotions, encourage self-expression through art creation, and provide cognitive-behavioral therapeutic advice in both verbal and visual ways. The project highlights the human- centered approach and investigates the potential of generative AI as an effective agent for conducting art therapy.

Download here

Assessing Social Media Users’ Affective Engagement with Narratives of Invisible Disability

Published in Conference of the Association for the Advancement of Affective Computing, 2023

Abstract : Narratives about invisible disabilities are poorly represented in public discourse and often go undisclosed, leading to false assumptions, discrimination, and stigma against those who experience these conditions. To address these issues, recent studies have suggested that disclosure of first- person narratives of invisible disabilities should be increased. To understand the mechanisms affecting recipients of such narratives, the present study evaluates how social media users (N = 124) engage affectively with this content in a digitally mediated narrative-form intervention designed to reduce harmful assumptions against persons who experience invisible disabilities. Results of this study indicate that such an intervention may prove effective at reducing harmful assumptions on the basis of visual cues, and in line with past research, finds that affect may play an important role in assumption-making processes. Findings from this study may be used to inform novel digital interventions capable of counteracting harmful assumptions that drive prejudicial behaviors against a wide range of populations and communities.

Download here

Empowering Learners with a Low-Barrier Mobile Data Science Toolkit

Published in FabLearn/Constructionism, 2023

Abstract : This paper introduces a novel data science toolkit designed specifically for children, enabling them to effortlessly create mobile apps integrated with data science capabilities. The toolkit showcases new features that simplify the data science process for young users. Additionally, the paper presents a collection of example apps created using the toolkit, highlighting the versatility and potential of this innovative platform. By empowering children to explore data science through app development, this toolkit opens exciting opportunities for hands-on learning and creative expression in the field of citizen science.

Download here

Exploring Prompt Engineering for Generative AI-Based App Generation

Published in Acta Scientific Computer Science, 2023

Abstract : This paper presents a platform that allows students to generate mobile applications for smartphones and tablets using natural language descriptions. We explore three methods of modifying the model’s input (prompts) to optimize the generated apps. We evaluate the model’s performance using the BLEU score and found that appropriate example pair selection and variation of the number of example pairs improve the quality of the generated apps. Finally, we discuss the implications for computer science education in light of generative models for code.

Download here

Learn and using Image Classifiers by Creating Real Mobile Applications

Published in 17th International Technology, Education and Development Conference, 2023

Abstract : The study discusses a learning platform created using MIT App Inventor and Personal Image Classifier (PIC) to teach image classification to students. The platform aims to make learning about and using image classification models easier and more enjoyable for students. Three computer science students from The Hong Kong Polytechnic University participated in the study and successfully learned and implemented the concepts of image classification in practical software prototypes. The study shows that the platform can enhance students’ interest in Artificial Intelligence and make them want to delve deeper into the mechanics of machine learning. The study concludes that the platform has the potential to improve computer science education by allowing students to apply machine learning models to solve real-world problems in their daily lives or the community.

Download here

Speak your mind : Introducing APTLY, the software platform that turns ideas into working apps

Published in 15th annual International Conference of Education, Research and Innovation, 2022

Abstract : MIT Aptly is a tool that automatically generates mobile apps from written or spoken natural language descriptions using OpenAI’s Codex. It lets people create programs without coding or knowledge of programming, and its app generation is based on example pairs and few-shot prompts. Aptly’s performance depends on the input given to OpenAI’s Codex, and it poses challenges for research in computational thinking education. The presentation will demonstrate Aptly’s preliminary performance and review its implementation.

Download here

Using Semantics of Textbook Highlights to Predict Student Comprehension and Knowledge Retention

Published in Third Workshop on Intelligent Textbooks, Springer., 2021

Abstract : This study uses students’ highlights in textbooks to predict their performance on quiz questions, and constructs a semantic representation using deep-learning sentence embedding technique (SBERT) to capture content-based similarity. We built regression models that include highlighting features and found that they reliably boost model performance. The highlighting features improved models for questions at all levels of the Bloom taxonomy. However, the generalization was not strong for held-out questions.

Download here

Inferring student comprehension from highlighting patterns in digital textbooks: An exploration in an authentic learning platform.

Published in Second Workshop on Intelligent Textbooks, Springer., 2020

Abstract : The study examines whether student comprehension and knowledge retention can be predicted from the material they choose to highlight in their textbooks. We found that the specific pattern of highlights made by students can explain about 13% of the variation in quiz scores. A low-dimensional logistic principal component based vector was the most effective input for a ridge regression model. Overall, highlights provide a strong signal of a student’s knowledge state.

Download here

David Young-Jae Kim

Publications