Reproducible, AI-assisted modeling and data science (MDS) projects

andreashandel.com

Andreas Handel

2026-09-08

Overview

  1. Concept — organized, reproducible, and AI-ready projects
  2. Implementation — a setup/template that turns those ideas into a practical working environment
  3. Hands-on — using the setup to start and implement your own project

Concept

Concepts of reproducible, AI-assisted project workflows

Motivation

An XKCD comic about reproducibility and the replication crisis.

xkcd.com · CC BY-NC 2.5
  • Reproducibility is a hallmark of good science
  • A well-organized, reproducible project makes work more efficient
  • AI-supported work is the future present

Problems with “standard” workflows

A cartoon showing an unexplained step labeled 'then a miracle occurs' in a mathematical derivation.

Sidney Harris · ScienceCartoonsPlus.com
  • Results drift from their code — Copied tables and figures become hard to update or verify
  • Files lose their roles — Raw, processed, current, and obsolete versions become hard to tell apart
  • Decisions disappear — Cleaning and modeling choices stay in memory or private conversations
  • Sharing becomes risky — Private data and unfinished material move into unsuitable places
  • Reproducibility is hard — Others, including your future self, will struggle to redo your work

The ORDERly concept

Four colour-coded boxes -- Organized (predictable places), Recorded (changes and decisions), Documented (purpose and instructions), and Executable (saved steps that run) -- add up, via an arrow, to a fifth box: Reproducible, a stated result can be recreated.

ORDERly AI work

The same four-box layout as the previous slide, with Restricted (privacy and scope limits) and Evaluated (checks before acceptance) marked by dashed rules to show they replace Recorded and Executable. They add up to Reproducible: bounded, inspectable, and project-owned.

Implementation

A setup to support an AI-assisted ORDERly workflow

An ORDERly project setup

A GitHub template repository: github.com/andreashandel/mds-project-template

Organized
The template provides a detailed file and folder structure to organize the project
Recorded
Git/GitHub history, generated outputs, AI-use logs, and project notes preserve the path taken
Documented
Markdown and Quarto files, folder READMEs, documented code, and AI policy explain what to do and why
Executable
Scripts and Quarto files regenerate processed data, analyses, figures, tables, and products
Reproducible
A modeling and data analysis project that can be reproduced from the raw data to all finished products.

Specific tools (largely interchangeable)

  • Editor/IDE
    Positron, VS Code, RStudio, JupyterLab
  • Coding language
    R, Python, Julia
  • Authoring
    Quarto, Typst, R Markdown, Jupyter Book
  • Tracking
    Git/GitHub, GitLab, Bitbucket, SVN
  • Agentic AI
    ChatGPT Codex, Claude Code, Gemini CLI
  • Workflow management (optional)
    renv, targets, Docker, Conda, Make

The project structure gives each part a role

The main path runs data/code/results/products/; the other folders and root files provide stable inputs and guidance.

A row of four boxes joined by arrows: data (inputs, derived data), code (workflow stages), results (generated outputs), and products (deliverables for the audience). Below sit three supporting boxes: assets (stable support), ai (policy, summary, log), and root files (readme.md, usage.md).

Separate raw, derived, private, and large data

A PhD Comics cartoon about whether data is singular or plural.

Jorge Cham · phdcomics.com
  • raw-data/ — Original input; preserve it unchanged
  • processed-data/ — Cleaned or transformed data generated by code
  • private-data/ — Restricted material kept in an approved location
  • large-files/ — Material too large for ordinary Git and GitHub use

Track changes with Git, guard what you share

An XKCD comic in which someone explains that the way to use Git is to memorize shell commands and start over when errors appear.

xkcd.com · CC BY-NC 2.5
  • Commit small, meaningful changes; push to back up and to share
  • Start private; go public later after checking data, outputs, license, and authorship
  • Keep sensitive data out via .gitignore
  • Nothing above roughly 20 MB; use the ignored large-files/ folders instead

Code follows the stages of the analysis

Each stage should have recognizable inputs, outputs, and instructions.

Four boxes joined by arrows: data-processing (raw to processed), data-exploration (checks, summaries), modeling-analysis (models, results), and figures-tables (final outputs). A wide box below reads utilities: optional support for repeated helper logic.

Keep generated outputs separate from everything else

results/
Code-generated model objects, intermediate outputs, figures, and tables
products/
Reports, manuscripts, supplements, presentations, posters, and apps
assets/
References, PDFs, schematics, and other stable supporting material
ai/
AI policy, project context, guidance, and meaningful-use records

Running the workflow

  1. code/data-processing/processing-code.rdata/processed-data/
  2. code/data-exploration/eda-code.rresults/output/
  3. code/modeling-analysis/statistical-analysis.rresults/output/
  4. code/figures-tables/make-tables.rresults/tables/
  5. code/figures-tables/make-figures.rresults/figures/
  6. quarto render the products you need → products/

Never hand-edit raw data, or results/products — change the code and rerun.

The template ships a working example

Run it end to end once, then replace it piece by piece with your own project.

Data and code
data/raw-data/example-data.xlsx and five short R scripts, one per workflow stage
Generated results
results/output/, results/tables/, and results/figures/, all rebuilt from code
Finished products
report.qmd, manuscript.qmd with supplement, an example presentation, and a poster stub

Every number and figure in those products can be traced back to a script.

AI can support project work

  • Code
    Write, document, refactor, and debug project code
  • Literature
    Help search, summarize, and organize background material
  • Writing
    Draft or improve reports, manuscripts, and presentations
  • Documentation
    Keep code, outputs, and documentation aligned

How AI plugs into this project

Run it inside the repository
Use an agentic tool that can read and write project files and run code
Point it at the documents first
readme.md, usage.md, agents.md, and code-guidelines.md carry the project rules
Bound what it may do
ai/ai-use-policy.md states what AI may touch, and what must never leave the project
Let it record its work
ai/ai-use-log.md is written by the AI; ai/project-summary.yml keeps it oriented
Keep changes reviewable
Ask for small steps, then rerun affected scripts and re-render affected products

AI can support ORDERly habits

Things worth asking your assistant to do:

Organized
“Check that every file sits in the folder its role calls for.”
Recorded
“Summarize what changed today and why, and log your own contribution.”
Documented
“Update the READMEs and usage notes to match what the code now does.”
Executable
“Run this script in a clean session and fix what breaks.”
Reproducible
“Trace this figure back to the data it came from and show me the path.”

An ORDERly project is easy to work with

  • Current you can work efficiently
  • Future you can recover the reasoning
  • Collaborators can inspect and extend the work
  • AI can assist appropriately
  • Others can understand and recreate the stated result

Hands-on

Getting started and using the setup/template

Getting started

  • Visit the template repository: github.com/andreashandel/mds-project-template
  • Read the readme file on the main page
  • Follow the instructions provided in the readme and further documents
  • Update the guidelines as needed
  • Involve AI early on and make sure all is documented

More information

Inside the template
readme.md · usage.md · new-project-instructions.md
MDS tools mini-courses
andreashandel.github.io/mds-tools
Questions and contact
andreashandel.com