GUIDED PROJECT INTERMEDIATE EXPERIMENTATION

Checkout A/B Test ship or not?

The checkout experiment finished Friday. Conversion is up, but is it safe to ship? You'll run a real A/B test analysis on a 14-day, 10k-session experiment, segment by device, and ship a Streamlit tool the PM can use without opening a notebook.

TOOLS & TECHNIQUES

Python pandas SciPy Statsmodels A/B testing Streamlit

▶ Start chapter 1

Google Meta

I've personally interviewed hundreds of data candidates.

Pasquale Bruno Founder · datascienceportfol.io Read bio

⏱

7–9 hours

self-paced

◉

8 chapters

video + written

◱

Synthetic dataset

10k sessions · 14-day experiment

✦

Built with Claude

Ch 6: README & LinkedIn post, grounded in your results

→

Live deployed

Streamlit Community Cloud

WHAT YOU'LL WALK AWAY WITH

Real, portfolio-ready artifacts, not a certificate.

✓

Analysis notebook

✓

Experiment summary doc

✓

Streamlit results tool

✓

Portfolio case study page

THE SCENARIO

The question isn't 'did it work'. It's 'should we ship'.

**Finch** is a UK DTC footwear brand (AOV ~£110). They ran a 14-day A/B test on their checkout flow: single-page redesign, Apple/Google Pay, trust badges, autofill. Conversion is up overall and statistically significant, but **the effect is entirely driven by desktop**. Mobile shows nothing.

You'll run the full analysis (10k sessions): an SRM check, a two-proportion z-test with confidence intervals, and a **segment deep-dive by device**, then write the experiment summary doc in the format Airbnb and Spotify actually use. You'll build a **Streamlit tool** that turns the findings into a self-serve dashboard, and use **Claude** to draft a README and LinkedIn post grounded in your actual results.

BY THE END, YOU'LL BE ABLE TO

✓

Run a clean two-proportion z-test for conversion lift, with a 95% confidence interval

✓

Check for Sample Ratio Mismatch (SRM) before trusting any downstream result

✓

Segment a significant result by device and recognise when an overall lift is masking a segment-level split

✓

Translate a p-value and confidence interval into a monthly revenue estimate a PM can act on

✓

Write an experiment summary doc in the format real product teams use, and use Claude to draft a grounded README and LinkedIn post from your actual findings

PROJECT CURRICULUM

8 chapters · ~7–9 hours

CLAUDE API NEW 13 videos 8 written lessons

Project brief & setup

video + reading

45 min +

● Read the brief: a 14-day, 10,000-session checkout A/B test. Is the lift real enough to ship?

● Set up Colab, Google Drive, and a GitHub repo with the right folder structure

● Run five sanity checks on the dataset, including the 9,460-null payment_method column

● Chapter checkpoint: data loaded, validated, and committed

Meet the data

video + code

55 min +

● Chart daily traffic across the 14-day window to rule out contamination

● Verify device and country splits are balanced between control and treatment

● Establish the headline baseline: control ~4.7% vs treatment ~6.1% conversion

● Explore order value distribution, checkout time, and the shift toward Apple Pay / Google Pay

Conversion rate test

video + code

55 min +

● Run a Sample Ratio Mismatch check before trusting any downstream result

● Run a two-proportion z-test: z=3.10, p=0.002, statistically significant

● Calculate a 95% confidence interval for the lift: [+0.51pp, +2.29pp]

● Translate the result into a monthly revenue estimate (£32k point estimate, £12k conservative)

Segment deep-dive (mobile vs desktop)

video + code

55 min +

● Break the aggregate result down by device: mobile vs desktop

● Find that desktop drives the entire effect (+2.49pp, p=0.006) while mobile shows nothing (+0.88pp, p=0.106)

● Learn why segmenting a significant result is not optional

● Write the real recommendation: ship to desktop, hold on mobile

Build the Streamlit tool

video + code

1h 50m +

● Build a single-file Streamlit app a PM can use without opening a notebook

● Wire up file upload, validation, and live recalculation of every metric

● Render metric cards, a conversion chart, and a device breakdown table

● Auto-generate the shipping recommendation from the significance flags

● Deploy it live to Streamlit Community Cloud with a public URL

Experiment summary doc 🧠 CLAUDE API

video + template

55 min +

● Write the experiment summary in the standard format used at Airbnb and Spotify

● Structure it as hypothesis → setup → results → interpretation → recommendation → next steps

● Use Claude to generate a styled PDF version with ReportLab, grounded in your real numbers

Ship it 🧠 CLAUDE API

video + code

45 min +

● Use Claude to draft a README from your actual experiment findings, then edit it into your own voice

● Use Claude to draft a LinkedIn post announcing the project

● Push the notebook, Streamlit app, and experiment summary doc to GitHub

Frame it for employers

reading

22 min +

● Memorise the four numbers you need cold before any interview

● Practice the 'walk me through this project' answer: finding first, not tool list first

● Learn how to explain why a significant aggregate result can still be an incomplete answer

● No code, just a reading chapter to prep you for interviews and portfolio review

▶ Start chapter 1

PREVIOUSLY

Google Meta

YOUR INSTRUCTOR

Pasquale Bruno Founder · datascienceportfol.io

I spent 10+ years in data and product roles at Google and Meta, including hundreds of hiring-panel interviews, so I know exactly what makes a hiring manager keep reading. I design and review every guided project myself, using briefs and datasets modelled on real analyst work, not toy CSVs. If you get stuck, you're emailing me directly, not a support queue.

10+ yrs

IN INDUSTRY

500+

CANDIDATES INTERVIEWED

300+

PORTFOLIOS REVIEWED

See all projects Connect on LinkedIn

SKILLS YOU'LL PRACTICE

Tools & techniques

Python pandas SciPy Statsmodels A/B testing Streamlit Two-proportion z-test Effect size Segment analysis Experiment summary docs Claude API

PREREQUISITES

Before you start

Comfortable with Python (functions, dicts, simple files) REQUIRED

Some pandas (DataFrames, groupby) REQUIRED

Awareness of basic stats (p-value, confidence interval), which we re-cover REQUIRED

Free GitHub + Streamlit Community Cloud accounts REQUIRED

Familiarity with e-commerce business model HELPFUL

🧠 NEW · USE AI LIKE A REAL JOB

Ship analysis with the Claude API in the loop.

Real data teams aren't writing SQL one query at a time anymore. A whole chapter teaches you to wire Claude into a real analysis, the way analysts at Stripe, Anthropic and Notion actually work.

README grounded in your real findings

Claude drafts the README from your actual experiment numbers, not a generic template

LinkedIn post, same approach

Announce the project publicly with a draft built from your real results

You edit, you own it

Claude structures what you already know. You review and add your own voice before publishing

Pricing

One plan gets you every Guided Project.

Free builds your portfolio. Job Ready adds every Guided Project on top, live now and every one we ship next.

Free

$0/forever

For anyone with projects ready to share.

✓ Portfolio website with unlimited projects
✓ Personalised URL
✓ No design nor technical skills needed
✓ All your education and experience info in one place

Start free

Job Ready

$99/ month

For people investing in a data career.

✓ Portfolio website builder PRO
✓ All current + future Guided Projects
✓ Discord support channel for any questions, answered by me
✓ Weekly live office hours with me
✓ 1:1 monthly call with me (first 20 members)
✓ Async portfolio & LinkedIn reviews (first 20 members)
✓ Cancel anytime

Get Job Ready

Not sure if it's right for you?

Book a free 15-minute call with me. We'll talk through where you are and whether a Guided Project makes sense.

Book a call →

HOW WE'RE DIFFERENT

Why this isn't another course.

FEATURE THIS PROJECT DATACAMP COURSERA

Real business scenario, not toy data ✓ ✕ ✕

Portfolio-ready deliverable ✓ ✕ –

Live deployed self-serve tool ✓ ✕ ✕

Claude drafts your README, grounded in your real results ✓ ✕ ✕

Certificate of completion ✓ ✓ ✓

Thousands of micro-lessons ✕ ✓ ✓

FREQUENTLY ASKED

Questions, answered.

Do I need a stats degree? +

No, we re-cover what you need (p-value, confidence interval, effect size). The harder skill is judgement, and that's what the project teaches.

Is the dataset real? +

It's synthetic but real-shaped, and the baked-in result mirrors a common production pattern (overall lift driven by one segment).

Do I need to deploy anything? +

Streamlit Community Cloud is free; we walk through it.

Can I cancel anytime? What about refunds? +

Yes, cancel anytime, no lock-in, right from your account. We don't offer refunds on subscription payments already made, but you'll keep full access through the end of the billing period you've already paid for.

Can I use this in interviews? +

Yes. A/B test readouts are one of the most-asked DS case-study formats.

CONTINUE YOUR PATH