Using Codebooks to Detect Cybercrime Topics in Text Narratives

The proposed contribution is a generalist-oriented prompting method that uses qualitative codebooks to detect cybercrime incidents in consumer narratives with pretrained language models.

Source: E003, E008

For impostor scams, the codebook condition achieved strong average precision and recall and improved average precision relative to the baseline.

Source: E010

For identity theft, baseline precision was weaker, while the codebook condition attained stronger average precision and recall and improved average precision over baseline.

Source: E006

In the reported case studies, every evaluated model apart from the identified exception surpassed the stated precision-and-recall threshold under codebook prompting.

Source: E003, E009

Generalization may be limited to cybercrime attributes or events that are amenable to qualitative analysis and codebook development.

Source: E005

The study evaluates only a single prompt template and does not establish whether more elaborate templates would improve performance.

Source: E016

The findings suggest that resource-constrained organizations may be able to use pretrained models, potentially including lower-cost options, without specialized model development or domain expertise.

Source: E016

The proposed contribution is a generalist-oriented prompting method that uses qualitative codebooks to detect cybercrime incidents in consumer narratives with pretrained language models.

Source: E003, E008

The findings suggest that resource-constrained organizations may be able to use pretrained models, potentially including lower-cost options, without specialized model development or domain expertise.

Source: E016

The research examines whether general-purpose pretrained language models can identify cybercrime in narrative text when guided by researcher-authored coding guidance.

Source: E004

The proposed contribution is a generalist-oriented prompting method that uses qualitative codebooks to detect cybercrime incidents in consumer narratives with pretrained language models.

Source: E003, E008

The empirical scope covers impostor scams and identity theft as the studied detection topics.

Source: E014

The approach assumes that its codebooks have been rigorously developed so that independent human annotators can apply them reliably and consistently.

Source: E001

The impostor-scam corpus comprises deduplicated complaint narratives and includes both positive and negative labels.

Source: E017

The identity-theft corpus comprises deduplicated complaint narratives and includes both positive and negative labels.

Source: E018

The datasets were derived from publicly available, privacy-scrubbed consumer narratives whose submitters had consented to public release.

Source: E007

The codebook condition uses a shared prompt structure containing a classification instruction, topic-specific coding guidance, a complaint narrative, and a constrained binary response.

Source: E002, E013

The baseline uses the same prompt structure but omits the coding guidance.

Source: E012

The evaluation compares codebook and baseline prompts across language models from the Gemini and GPT families, with repeated runs and generally default settings where feasible.

Source: E015

Prompt performance is assessed using precision and recall.

Source: E015

For impostor scams, the codebook condition achieved strong average precision and recall and improved average precision relative to the baseline.

Source: E010

For identity theft, baseline precision was weaker, while the codebook condition attained stronger average precision and recall and improved average precision over baseline.

Source: E006

In the reported case studies, every evaluated model apart from the identified exception surpassed the stated precision-and-recall threshold under codebook prompting.

Source: E003, E009

Development of the impostor-scam codebook included an inter-annotator agreement assessment on a shared narrative set, with strong reported agreement.

Source: E017

Generalization may be limited to cybercrime attributes or events that are amenable to qualitative analysis and codebook development.

Source: E005

The study evaluates only a single prompt template and does not establish whether more elaborate templates would improve performance.

Source: E016

Public availability of a detector and its codebook could enable users to alter narratives strategically in order to influence classification outcomes.

Source: E005, E011

The findings suggest that resource-constrained organizations may be able to use pretrained models, potentially including lower-cost options, without specialized model development or domain expertise.

Source: E016