{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# COAE Practical 04 — Membership-Inference Evaluation\n",
        "**Track:** AI Privacy | **Level:** Medium\n",
        "\n",
        "## Mission\n",
        "You are auditing a model in a **synthetic offline privacy experiment**. An adversary receives a confidence score for each record and wants to infer whether that record was included in training.\n",
        "\n",
        "You are given labelled *shadow evaluation data*. Pick a threshold so your membership predictor achieves **at least 75% true-positive rate** while keeping the **false-positive rate at or below 25%**.\n",
        "\n",
        "These scores are fabricated and do **not** reveal any real person's private information. This is a threshold-based pedagogical approximation; real membership inference needs carefully designed controls and reference distributions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "MEMBER_CONF = [0.97, 0.92, 0.84, 0.78, 0.76, 0.74, 0.70, 0.65]\n",
        "NONMEMBER_CONF = [0.85, 0.73, 0.69, 0.62, 0.58, 0.55, 0.44, 0.41]\n",
        "\n",
        "def evaluate_threshold(threshold):\n",
        "    tp = sum(s >= threshold for s in MEMBER_CONF)\n",
        "    fn = len(MEMBER_CONF) - tp\n",
        "    fp = sum(s >= threshold for s in NONMEMBER_CONF)\n",
        "    tn = len(NONMEMBER_CONF) - fp\n",
        "    return {\n",
        "        \"TPR\": tp/(tp+fn),\n",
        "        \"FPR\": fp/(fp+tn),\n",
        "        \"precision\": tp/(tp+fp) if tp+fp else 0,\n",
        "        \"tp\": tp, \"fp\": fp, \"tn\": tn, \"fn\": fn\n",
        "    }\n",
        "print(\"Baseline example:\", evaluate_threshold(0.90))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Task A — Calibrate membership inference\n",
        "You may set the threshold only, or implement a small threshold search. Do not edit the member and nonmember observations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def choose_threshold():\n",
        "    # TODO: Choose a threshold meeting BOTH conditions.\n",
        "    # TPR >= 0.75 and FPR <= 0.25\n",
        "    return 0.90\n",
        "\n",
        "threshold = choose_threshold()\n",
        "results = evaluate_threshold(threshold)\n",
        "print(\"Threshold:\", threshold)\n",
        "print(results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Task B — Verification\n",
        "The checker will fail until both the true-positive and false-positive requirements are met."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "assert isinstance(threshold, (int,float)) and 0 <= threshold <= 1, \"Use a valid probability threshold.\"\n",
        "assert results[\"TPR\"] >= 0.75, \"Too many actual members were missed.\"\n",
        "assert results[\"FPR\"] <= 0.25, \"Too many nonmembers were incorrectly flagged.\"\n",
        "print(\"PASS — membership inference operating point meets both constraints.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Task C — Privacy assessment\n",
        "Explain why the false-positive rate matters, how an adversary could abuse membership inference, and why confidence gaps are not universally reliable.\n",
        "\n",
        "**Bonus:** Evaluate a range of thresholds and report the resulting TPR/FPR trade-off. Discuss whether changing a model's training regimen or applying differential privacy could reduce membership information leakage.\n",
        "\n",
        "**Deliverable:** Threshold, checker output, limitations, a brief privacy-impact statement and mitigation recommendations."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python (Pyodide)",
      "language": "python",
      "name": "python"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}