Using Codebooks to Detect Cybercrime Topics in Text Narratives

The proposed contribution is a generalist-oriented prompting method that uses qualitative codebooks to detect cybercrime incidents in consumer narratives with pretrained language models.

Source: [3], [8]

The findings suggest that resource-constrained organizations may be able to use pretrained models, potentially including lower-cost options, without specialized model development or domain expertise.

Source: [16]

The research examines whether general-purpose pretrained language models can identify cybercrime in narrative text when guided by researcher-authored coding guidance.

Source: [4]

The empirical scope covers impostor scams and identity theft as the studied detection topics.

Source: [14]

The approach assumes that its codebooks have been rigorously developed so that independent human annotators can apply them reliably and consistently.

Source: [1]

The impostor-scam corpus comprises deduplicated complaint narratives and includes both positive and negative labels.

Source: [17]

The identity-theft corpus comprises deduplicated complaint narratives and includes both positive and negative labels.

Source: [18]

The datasets were derived from publicly available, privacy-scrubbed consumer narratives whose submitters had consented to public release.

Source: [7]

The codebook condition uses a shared prompt structure containing a classification instruction, topic-specific coding guidance, a complaint narrative, and a constrained binary response.

Source: [2], [13]

The baseline uses the same prompt structure but omits the coding guidance.

Source: [12]

The evaluation compares codebook and baseline prompts across language models from the Gemini and GPT families, with repeated runs and generally default settings where feasible.

Source: [15]

Prompt performance is assessed using precision and recall.

Source: [15]

For impostor scams, the codebook condition achieved strong average precision and recall and improved average precision relative to the baseline.

Source: [10]

For identity theft, baseline precision was weaker, while the codebook condition attained stronger average precision and recall and improved average precision over baseline.

Source: [6]

In the reported case studies, every evaluated model apart from the identified exception surpassed the stated precision-and-recall threshold under codebook prompting.

Source: [3], [9]

Development of the impostor-scam codebook included an inter-annotator agreement assessment on a shared narrative set, with strong reported agreement.

Source: [17]

Generalization may be limited to cybercrime attributes or events that are amenable to qualitative analysis and codebook development.

Source: [5]

The study evaluates only a single prompt template and does not establish whether more elaborate templates would improve performance.

Source: [16]

Public availability of a detector and its codebook could enable users to alter narratives strategically in order to influence classification outcomes.

Source: [5], [11]