Blog
Improving Behavior Science Experiments with LLMs and Milvus

Improving Behavior Science Experiments with LLMs and Milvus

May 31, 20247 min read

Pressing and Unpressing “Buttons” — Can We Reframe Our Emotions?

Absolutely. Dr. Damon Abraham, a behavioral scientist with a PhD in Psychology, has a research project demonstrating how reappraising an image can shift our valence and arousal emotions. Thus, the quest is to discover what in an image allows that shift and how to search for it. This is where multidimensional image embeddings and vector databases come into play.

In his presentation 'From Images to Meanings to Vector Databases’' at the Zilliz Unstructured Data Meetup in Seattle on February 13, 2024, Dr. Damon Abraham talks about his innovative research project aimed at creating a stimuli repository for experimental psychology using LLM and vector database technologies. This interdisciplinary initiative involves collaboration with the University of Denver and researchers from the Institute of Love and Time and the Institute of Noetic Sciences. It seeks to develop an open-source normative database of images and techniques to measure the dynamic psychological distance between images and their potential for a successful reappraisal.

Watch Dr. Damon Abraham’s Talk

The Psychology Project

Thousands of human raters evaluated 2,500 images from a normative database, measuring each for emotional intensity (arousal) and valence (positive vs. negative) across 18 distinct stimulus classes. The goal is to better understand the emotional meaning of these images by creating a normal distribution of responses to select specific images to trigger precise emotional reactions in behavioral studies.

Central to Dr. Abraham's work is the emotion regulation technique of cognitive reappraisal, which involves reframing the meaning of a stimulus to change its emotional impact.

According to the_ National Library of Medicine_, reappraisal is a commonly-used and widely-studied emotion regulation strategy that involves reframing the meaning of a situation to alter its emotional impact (Gross, 1998).

This cognitive reappraisal technique demonstrates that images can evoke particular cognitive and emotional responses, making it a valuable mechanism in psychological experiments. This research effectively shows how different contextual interpretations of images can change our feelings, providing profound insights into emotional regulation.

Cognitive reappraisal explores emotional response shifts from the natural unfolding. Let’s take the following car accident image as an example. At first sight, the viewer could be shocked and find it gruesome. On a second analysis, positive aspects might be perceived, such as no one hurt in the picture or even the interpretation of being a fabricated image rather than a factual tragedy. So, by changing how you think, you can change how you feel.

Taxi car damaged in a crash. Taxi car damaged in a crash.

If images can trigger cognitive and emotional reactions, then a normative database of images we can search based on stimuli similarity would be invaluable in psychological experiments.

Measuring Psychological Distance

A key insight from this project is that psychological distance should not be treated as a fixed measure because context heavily influences it. Having control handles that are dynamic can unleash advanced image processing use cases.

The research team employed Construal Level Theory (CLT) as a framework to understand and measure psychological distance.
According to Wikipedia, “Construal level theory (CLT) is a theory in social psychology that describes the relation between psychological distance and the extent to which people's thinking (e.g., about objects and events) is abstract or concrete_. The core idea of CLT is that the more distant an object is from the individual, the more abstract it will be thought of, while the closer the object is, the more concretely it will be thought of.”

The project highlights that mental representations fluctuate from proximal, here and now (e.g., direct sensory experience) to a more removed scenario in time, space, or even factuality (e.g., fictional image from the past). Essentially, as we transition from concreteness (low-level construal) to abstraction (high-level construal), the contextual network of related concepts augments, increasing the complexity of cognitive processing involved when evaluating an image.

Construals serve as cognitive mechanisms that modulate our perspective on psychological distance. When conceptualizing events or scenarios that are temporally or spatially distant, we tend to employ more abstract reasoning, focusing on the 'why' — the broader reasons, ideals, or principles involved. As these events draw nearer, our thinking shifts towards the more immediate and practical 'how' — the specific actions required for dealing with the situation.

This shift from abstract to concrete thinking aids in planning and managing future scenarios, enhancing our ability to navigate potential objections or challenges more effectively.

Getting the Right Data

Psychological distance is defined along several dimensions, with temporal, spatial, social, and hypothetical distances considered the most important.

As pointed out in the presentation, tools such as Google Vision can classify images. It does a great job of identifying what is in the image. However, it doesn't capture its deeper meaning—it can tell you the objects in the scene but not much about how a person might react to that image. Moving from the "what" to exploring the "how" is core to the project proposal.

In the study, thousands of participants from the University of Denver and Amazon Mechanical Turk were asked to rate each image across 18 distinct psychological dimensions, such as emotional, movement, embodiment, social, and physical distance, resulting in over 760,000 unique ratings.

This rich dataset provides a complete distribution of ratings, serving as a valuable resource within the framework of Construal Level Theory and other psychological research. It offers robust data that can be analyzed and interpreted through the lens of these theories, enabling researchers to gain insights into various dimensions of images and their psychological implications.

The rating provided each image with dimensions that offer additional perceptive information.

Leveraging the Power of LLMs and Vector Databases

The team began by extracting visual embeddings using the EfficientNet-B4 model to capture the images' semantic content and general composition.

They further enhanced the data using a multimodal model, LLaVA, to generate detailed descriptions (up to 256 tokens) of each image. These descriptions were then converted into vector embeddings using OpenAI’s text-embedding-ada-002 model.

The next step is to transfer the embeddings to a vector database, such as Milvus and Zilliz Cloud (the fully managed Milvus), to better handle and process its multidimensionality at scale.

This approach allows the team to measure and perform semantic and visual similarity searches on information embedded in each image, enhancing the ability to find images with similar emotional impacts or cognitive appeals.

With the image embeddings enriched with additional information and available in a vector database for fast analysis, they can be used in behavioral science, with vast applications in therapeutics, safety, product design, and marketing strategy, among other use cases.

Getting a Handle on Reappraisal Affordances

The study laid a foundation for understanding the emotional shifts caused by reappraisals. Participants were instructed to view an image, rate it, and then reassess it to see how their emotional responses changed. The findings revealed that people respond more negatively when they allow their emotions to unfold naturally.

However, beyond assessing the valence and arousal of images, it is intriguing to explore what triggers these emotional shifts and what enables individuals to change their reactions to an image. This exploration examines how different images "afford" or "restrict" opportunities for cognitive reinterpretation, termed 'reappraisal affordances.'

The concept of affordances, introduced by Gibson, describes how the characteristics of objects suggest their possible uses, thus influencing our interactions. In psychological terms, an image’s inherent semantic properties and the range of associations it invokes may affect its capacity for reappraisal.

The research seeks to determine how these properties correlate with an image's susceptibility to cognitive reappraisal, potentially expanding our understanding of how context and perception affect emotional regulation. For example, a snarling dog presents fewer opportunities for reappraisal due to its low-level construal and limited contextual scope. In contrast, an image of a Nazi, positioned within a high-level construal and embedded in a broader semantic network, offers greater affordances for reinterpretation.

Wrapping Up

Dr. Damon Abraham is pioneering an exploration into the emotional resonance of images, their capacity to evoke cognitive reappraisal, and how to instrument it with LLMs and a vector database like Milvus and Zilliz Cloud. His research uncovers the key components within image classes that can reliably predict their ability to facilitate emotional shifts, thus shedding light on the nuanced mechanisms underlying our responses to visual stimuli. His work opens a vast horizon of applications and use cases that leverage how images evoke emotional responses and influence decision-making processes.

Exciting, isn't it? Got ideas for an application? Get started!

If you wish to learn more or begin your project on constructing a multidimensional vector database of images for stimulus similarity search, for instance, welcome to join our Discord channel. We provide a wealth of resources and a supportive community to assist you in getting started.

Daniella Pontes

Content

Start Free, Scale Easily

Try the fully-managed vector database built for your GenAI applications.

Try Zilliz Cloud for Free