Spaces:

meta-llama

/

README

like 7

Running

Hello everyone,

I’m working on a research project where we use Llama 3.2 in R to analyze how trust is defined in academic studies. Our dataset consists of 106 studies on trust, UX, and conversational agents, forming part of a scoping review.

To classify how trust is addressed, we’ve defined four categories:
:small_blue_diamond: “cognitive” (trust based on logic, transparency, consistency, and rational engagement)
:small_blue_diamond: “emotional” (trust based on empathy, social connection, and human-like interaction)
:small_blue_diamond: “other” (trust that doesn’t fit cognitive or emotional)
:small_blue_diamond: “none” (no trust expressed)

We are using a few-shot prompt to improve classification accuracy, but Llama 3.2 still produces inconsistent and unreliable results. Sometimes, it misclassifies texts that clearly fall into cognitive trust as emotional or other. We have already adjusted the temperature, top_k, and top_p settings, but we are still seeing inaccuracies.

Could you help us refine the prompt to make it more robust and ensure more valid and reliable classifications? Any insights on prompt engineering for Llama models or experiences with similar text classification tasks would be greatly appreciated!

Here’s the relevant part of our R script:

library(xml2)
library(dplyr)
library(tibble)
library(purrr)
library(rollama)

:open_file_folder: Set file path for XML dataset
:pushpin: List all XML files
xml_files ← list.files(xml_folder, pattern = “.xml$”, full.names = TRUE)

:pushpin: Create DataFrame to store results
results ← tibble(Source_File = character(), Title = character(), DOI = character(), Trust_Type = character())

:pushpin: Few-shot examples
examples_fs ← tibble::tribble(
~text, ~answer,
“Users trust AI systems when they understand the logic behind their decisions.”, “cognitive”,
“People feel emotionally connected to chatbots that show empathy and personal engagement.”, “emotional”,
“Trust in AI assistants can arise when they provide consistent responses over time.”, “cognitive”,
“Users tend to trust chatbots when they feel that their emotions are understood.”, “emotional”,
“Trust in AI is sometimes driven by social norms and external influences rather than direct interaction.”, “other”,
“Some users are skeptical about chatbots and do not trust them at all.”, “none”
)

:pushpin: Loop through XML files
for (file in xml_files) {
doc ← read_xml(file)

title ← xml_text(xml_find_first(doc, “//tei:titleStmt/tei:title[@type=‘main’]”, ns)) %>% coalesce(“Unknown Title”)
doi ← xml_text(xml_find_first(doc, “//tei:idno[@type=‘DOI’]”, ns)) %>% coalesce(“No DOI”)
text_body ← xml_text(xml_find_all(doc, “//tei:body”, ns)) %>% coalesce(“No content available”)

q_fs ← make_query(
text = text_body,
prompt = "The following text will be analyzed to determine whether it expresses cognitive trust, emotional trust, another form of trust, or no trust at all.

Cognitive trust is based on logic, transparency, consistency, cognitive absorption, and rational engagement. It occurs when trust is built on understanding how a system works, its reliability, and its ability to provide predictable and explainable results.

Emotional trust is based on empathy, personal connection, and human-like interaction. It occurs when trust is built on feelings, social presence, and affective engagement.

Other forms of trust do not clearly fall into cognitive or emotional categories, and ‘none’ applies when trust is not expressed at all.

Please classify the following text using only one of these categories: ‘cognitive’, ‘emotional’, ‘other’, or ‘none’.",
template = “{text}\n{prompt}”,
system = "You analyze texts and determine whether they express cognitive trust (trust based on logic, consistency, transparency, cognitive absorption, and rational engagement), emotional trust (trust based on empathy, personal connection, and human-like interaction), another form of trust (‘other’), or no trust at all (‘none’).

Answer only with ‘cognitive’, ‘emotional’, ‘other’, or ‘none’.
Do not provide explanations.",
examples = examples_fs
)

response ← rollama::query(
q_fs,
model = “llama3.2”,
model_params = list(
seed = 12,
temperature = 0,
top_k = 1,
top_p = 0.1
)
)

trust_category ← response$message$content %>% coalesce(“none”)

results ← bind_rows(results, tibble(Source_File = file, Title = title, DOI = doi, Trust_Type = trust_category))
}

:pushpin: Print results
print(results)