Red Teaming Prompt Templates

Copy or adapt these Markdown prompts when asking an LLM to red-team a project plan or generated code. Sanitize the plan or code first: remove individual-level records, PHI, PII, credentials, private paths, controlled-access data, sensitive small-cell outputs, and protected system details.

The BMI examples below illustrate review questions, not clinical recommendations. Assess population mappings against the stated guideline and scope.

Red teaming involves adopting an adversarial perspective to identify potential weaknesses, assumptions, or edge cases in your project plans or code. These templates help you use LLMs as skeptical reviewers to strengthen your methodology and implementation.

## Template 1: Red Teaming Project Plans

Use this template to critically evaluate your project methodology before implementation:

```markdown
You are a skeptical methods reviewer with expertise in [INSERT DOMAIN, e.g., clinical epidemiology, biostatistics]. Review the project plan below:
~~~
<Insert sanitized project plan with schema and methods only>
~~~
Adopt a "not impressed" stance. List up to 10 missing assumptions, edge cases, or ambiguities that could be challenged by a critical reviewer. Pay particular attention to:
- Population-specific considerations and demographic heterogeneity
- Data quality issues inherent to the data source (e.g., EHR, survey, registry)
- Temporal considerations and measurement context
- Missing or ambiguous operational definitions
- Statistical validity and potential for bias

For each issue, write in 1-2 concise bullet points explaining why it could be problematic. Distinguish demonstrated problems from hypotheses; cite the relevant plan passage and propose a concrete check or decision that would resolve it.

If the plan appears robust and you find fewer than 10 issues, list as many as you can and briefly explain why the plan is otherwise comprehensive.
```

### Example Issues This Might Identify:
- **Methodological gaps**: Missing validation approaches, undefined outlier detection thresholds
- **Domain-specific oversight**: Failure to account for clinical contexts affecting measurements
- **Technical assumptions**: Presumptions about data standardization or temporal relationships
- **Definitional ambiguities**: Vague terms lacking operational definitions
- **Population heterogeneity**: Oversimplified demographic categorizations that ignore within-group variation or data quality issues

## Template 2: Red Teaming LLM-Generated Code

Use this template to critically evaluate code implementation:

```markdown
You are a skeptical code reviewer. Review the code below and identify potential issues:
~~~
<Insert LLM-generated code only; do not include real data rows or sensitive logs>
~~~
Adopt a "not impressed" stance. List up to 10 potential problems, including:
- Logical errors or edge cases not handled
- Performance inefficiencies
- Data quality assumptions that might not hold
- Missing validation steps
- Potential for incorrect results

For each issue, explain in 1-2 bullet points why it could be problematic. Cite the affected code and suggest a minimal synthetic test. Do not claim that a hypothetical bug was reproduced unless you ran the test and observed it.
```

### Example Issues This Might Identify:
- **Edge case handling**: Missing checks for empty datasets, malformed data, unexpected types
- **Performance issues**: Inefficient loops, unnecessary operations, memory problems
- **Domain-specific oversights**: Failure to handle measurement artifacts or collection biases
- **Statistical validity**: Inappropriate methods, missing assumptions, inadequate validation
- **Error propagation**: Early mistakes compounding into incorrect final results

## Tips for Effective Red Teaming

1. **Be specific about expertise**: Specify the domain expertise you want the LLM to adopt (e.g., "clinical epidemiologist," "biostatistician," "health informatics expert")
2. **Target known problem areas**: Direct attention to methodological areas prone to issues in your field
3. **Iterate**: Apply red teaming at different stages - initial plans, draft code, and final implementation
4. **Document responses**: Keep track of identified issues and how you addressed them
5. **Don't dismiss easily**: Even seemingly minor issues might reveal important oversights
6. **Consider intersectionality**: Ask about interactions between different methodological choices

## Domain-Specific Red Teaming Prompts

### For Health/Clinical Research:
```markdown
Focus particularly on:
- Population heterogeneity and demographic classification issues
- Age-stratified analysis requirements and developmental considerations
- Clinical context and measurement artifacts
- Temporal relationships and exposure windows
- Data provenance and collection biases
- Regulatory and ethical considerations
```

### For Observational Studies:
```markdown
Pay special attention to:
- Selection bias and missing data patterns
- Confounding and causal inference issues
- Measurement error and misclassification
- Time-varying exposures and outcomes
- Generalizability across populations
```

### Example: Population-Mapping Critique
A reviewer might ask: "Which guideline, version, purpose, and named populations support each proposed BMI cutoff? Show the mapping from the actual source variables to those populations; broad EHR race labels or combined categories do not establish that mapping. Specify how missing, refused, multiracial, and harmonized values are handled. For this synthetic adult tutorial, retain general descriptive categories and audit demographic missingness without generating population-specific clinical classifications."

### Example: Age-Stratified Analysis Critique
A reviewer might ask: "The tutorial restricts eligibility to age 20 or older at measurement. Show a check for the 20th birthday, the day before, and missing or invalid dates; do not use age at extraction. If `growthcleanr` is proposed, verify its supported adult use and settings for the installed version. A pediatric analysis requires a separately specified method and reference rather than reusing this tutorial's adult thresholds."

## Acting on Red Teaming Feedback

For each identified issue:
- Evaluate whether it represents a real risk to your analysis
- Modify your methodology or code to address legitimate concerns
- Add validation checks or error handling where appropriate
- Document assumptions or limitations that cannot be easily resolved
- Test edge cases that were previously overlooked

Use the review to prioritize concrete fixes and verification; a confident review alone is not evidence that the method or implementation is correct.