How to do data analysis in Python with pandas (and an AI helper)
The short answer
To do data analysis in Python, set up one environment per project, load risky columns as text, profile the data, clean it with tested functions, join tables with validate=, group with pandas, chart with a finding title, report effects with intervals, check AI-written code against independent totals, and re-run the notebook from a fresh kernel before sharing.
Want the tools, not just the method? The The Python for Data Analysis with an AI Helper Kit kit turns this guide into ready-to-use templates, spreadsheets and printables, with an instant download.
- Count the problems in an export before fixing any of them.
- Every merge gets validate= and a control total before and after.
- pandas 3 changed text types and copies: chained assignment never works now.
- Report the effect and its interval first, then the p-value.
- AI helpers write code; you decide whether the number is right.
- Only Restart & Run All proves a notebook works.
What are the steps of a data analysis in Python?
A reliable analysis in Python follows the same ten steps whatever the data. Each step has one check that stops a wrong number from travelling further. The steps below are the backbone of the method; the rest of this guide explains the ones people most often get wrong.
- 1. Write the question with a number in it, the decision it feeds and who decides.
- 2. Create one environment per project and pin package versions.
- 3. Load risky columns (money, dates, IDs) as text.
- 4. Profile: shape, types, duplicates, label variants, odd values.
- 5. Clean with small tested functions and a documented corrections file.
- 6. Join at the grain both tables share, with validate= and a control total.
- 7. Group, pivot or resample to answer the question.
- 8. Chart with the finding as the title and honest axes.
- 9. If you test, plan it first and report the effect and interval.
- 10. Restart & Run All, run the tests, then share a readable report.
How do I set up Python for data analysis, and which libraries do I need?
Fewer than most course lists suggest. pandas holds and transforms tables; NumPy sits underneath it and handles arrays and maths; matplotlib draws charts, and seaborn adds quicker statistical plots such as box plots and heatmaps on top of it; SciPy provides the statistical tests (two-sample t-tests, chi-square, the normal distribution); and Jupyter, through JupyterLab or VS Code, gives you notebooks that mix code, results and notes. pytest checks your own functions. That set covers loading, cleaning, joining, grouping, charting and simple testing for almost any business dataset.
Resist adding packages for every small task. Each one is another thing to install, pin, update and trust, and an AI assistant will happily suggest packages you do not need, or that do not exist. A short, pinned list in a requirements or lock file makes the analysis portable: a colleague, an auditor or you next year can recreate the same environment and get the same answer. Upgrade deliberately, once a quarter, after re-running your tests on the new versions.
Use one isolated environment per project, so the analysis runs the same way next month and on a colleague's laptop. uv creates an environment and records exact versions in a lockfile; conda (through Anaconda or Miniforge) also manages non-Python libraries; venv and pip come with Python; a hosted notebook such as Google Colab needs no install but its resources are not guaranteed and idle runtimes time out.
Check licences before you install. Anaconda's terms, dated 15 July 2025, require a paid Business plan for for-profit organisations with more than 200 employees or contractors; small companies, eligible academic institutions and eligible non-profits can use it free. If your notebook says pandas is not installed when pip says it is, your notebook kernel is a different Python: select the project environment as the kernel rather than installing again.
What changed in pandas 3 that analysts need to know?
pandas 3.0, released in January 2026, made three changes that break older code and many AI-generated snippets. Text columns now load as a dedicated str type, so code that checks dtype == object to find text misses them. Copy-on-Write is now the rule: any subset behaves as a copy, so chained assignment such as df[mask]['col'] = x never changes df, and the old SettingWithCopyWarning is gone. Change values with df.loc[mask, 'col'] = x instead.
Dates parsed from strings now default to microsecond resolution, so turning dates into integers gives numbers 1000 times smaller than before. pandas 3 also requires Python 3.11 or later. If you learned pandas from older tutorials, these are the first three things to unlearn.
How do I clean messy data in pandas?
Never edit the raw file and never fix values by hand inside the analysis. Load money, dates and IDs as text, count the problems (duplicates, test rows, text money, mixed date formats, label variants), then write one small function per recurring problem: normalise labels with a dictionary that raises an error for unknown values, parse each date format explicitly, convert money with to_numeric after stripping dollar signs and commas. One-off fixes go into a corrections file with the old value, new value, approver and reason.
Statisticians' rules for spreadsheet data (be consistent, write dates as YYYY-MM-DD, one value per cell, a data dictionary, no colour as data) double as a checklist of what a profile finds, and keeping data tidy, one variable per column and one observation per row, makes every later step simpler.
Why do totals go wrong after a pandas merge?
Because the lookup table has duplicate keys, or the two tables have a different grain. A left join onto a customer table with two rows for the same customer copies that customer's orders; joining orders to monthly spend on campaign alone attaches every month's spend to every order. merge has an argument for exactly this: validate states the expected relationship (one_to_one, one_to_many, many_to_one), and pandas raises a MergeError if the keys break it; indicator=True shows which rows matched. Add a row count and a total before and after every join.
| Step (SAMPLE data) | Rows | Revenue (USD) |
|---|---|---|
| Clean orders | 5,621 | 457,184.92 |
| AI draft: orders + raw CRM export (no validate) | 6,064 | 493,658.75 |
| Difference | +443 | +36,473.83 (+8.0%) |
| Reviewed: deduplicated CRM, validate='many_to_one' | 5,621 | 457,184.92 |
How should I use an AI assistant to write Python code?
Let it draft and explain; do not let it decide that a number is right. Studies point in different directions: in one controlled experiment developers finished a single JavaScript task 55.8% faster with an AI pair programmer, while a randomised trial with 16 experienced open-source developers found early-2025 tools increased completion time by 19% even though the developers believed they had been sped up. Code assistants also invent package names; one study found hallucinated packages averaged at least 5.2% for commercial models and 21.7% for open-source models. Check every package name before installing.
A six-step loop keeps the numbers right either way: give context (versions, column names, dtypes, fake rows), ask for small steps with explanations, read every line, run it yourself on the real data from a fresh kernel, check the output against an independent total or a test, and log it. Do not paste personal or confidential data into tools your organisation has not approved.
Reading helps; measuring tells you what to work on. These AI-graded assessments on AssessAll pair with this topic:
How do I handle dates and time series in pandas?
Parse dates with an explicit format, then use the .dt accessor for years, months and weekdays, and resample for periods: resample('W-SUN') gives weeks ending Sunday, resample('MS') gives calendar months, and rolling(7).mean() gives a seven-day moving average. Use periods (to_period('M')) when you report by month, because every March order then carries the same label and month arithmetic is exact, which is what cohort tables need.
Watch for zeros that mean 'not recorded'. In the worked example, a broken analytics tag reported zero sessions for three June days while orders continued. Counting all June orders against the recorded sessions gave a conversion rate of 2.39%; using only the orders from days whose sessions were recorded gave 2.17%, in line with the rest of the year. Mark such zeros as missing, compare like with like in every ratio, and write the gap into the method note. If your data has times, convert to the local time zone before cutting days, or events near midnight land on the wrong date.
How do I report an A/B test result honestly?
Plan the sample size and the reading date before the test starts, then lead with the effect and its interval, and give the p-value last. The American Statistical Association's statement on p-values is clear that a p-value does not measure the probability that a hypothesis is true or the size of an effect, and that decisions should not rest on a threshold alone. In the worked example, a one-page checkout converted at 4.60% against 4.00%: a difference of +0.60 points, 95% interval +0.04 to +1.15, p = 0.034. The honest summary: B probably converts better, and the lift could be anywhere from almost nothing to about 1.1 points.
When the interval includes zero, say 'we cannot tell yet', not 'it did not work'. A small charity test with about 1,200 recipients per email gave p = 0.47; detecting a 0.5-point change from 2.9% would need about 19,153 recipients per version.
What are the most common mistakes in Python data analysis?
Most wrong numbers come from a short list of habits, and each has a simple fix.
- Summing a column that is still text: pandas 3 returns a long string, not an error. Check df.dtypes first.
- Merging without validate= and a row count.
- Treating zeros from a broken tracker as real values: mark them missing.
- Fixing values by hand instead of in code or a corrections file.
- Reporting 'significant' without the effect and its interval.
- Sharing a notebook that has only been run cell by cell: in a study of more than a million GitHub notebooks, only 24.11% of those re-run executed without errors.
- Accepting AI-written code because it runs.
How do I make a notebook reproducible and shareable?
Restart the kernel and run all cells before every share. Keep a project folder with data, src, notebooks, tests and outputs; move logic that matters into functions with tests; record package versions; and use version control. The UK Government Analysis Function's reproducible analytical pipeline approach asks for the same: minimal manual steps, open-source tools, version control, peer review and documentation.
For readers, export the notebook as HTML with the code hidden, or copy the key outputs into a one-page report with a finding title, key numbers, the decision needed and a method note. The Python for Data Analysis with an AI Helper Kit includes five practice projects, a readiness check and a report template for this step.

Practise on messy data with known answers
The Python for Data Analysis with an AI Helper Kit gives you five end-to-end projects, tested code, a verification loop for AI output and the templates to run it at work.
Sources
- pandas project (official documentation): What's new in 3.0.0
- pandas project (official documentation, pandas 3.1.1): pandas.merge
- Anaconda, Inc. (official terms): Anaconda Terms of Service, Section 1
- Google (official FAQ): Google Colab Frequently Asked Questions
- UK Government Analysis Function: Reproducible Analytical Pipelines (RAP)
- Journal of Statistical Software: Wickham, H. Tidy Data. JSS 59(10), 1-23
- PeerJ Preprints (preprint v2, not the peer-reviewed text; journal version: The American Statistician 72(1), 2-10, 2018): Broman, K. W. and Woo, K. H. Data Organization in Spreadsheets (preprint)
- Empirical Software Engineering (Springer): Pimentel, J. F., Murta, L., Braganholo, V. and Freire, J. Understanding and improving the quality and reproducibility of Jupyter notebooks. EMSE 26, 65
- American Statistical Association (release of the statement; full text in The American Statistician 70(2), 129-133): ASA Statement on Statistical Significance and P-Values (six principles); Wasserstein, R. L. and Lazar, N. A.
- arXiv (Microsoft Research, GitHub and MIT Sloan authors): Peng, S., Kalliamvakou, E., Cihon, P. and Demirer, M. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
- arXiv (METR): Becker, J., Rush, N., Barnes, E. and Rein, D. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- USENIX Security Symposium 2025: Spracklen, J. and colleagues. We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
More from the Bodhih family
Questions people ask next
Is pandas still worth learning in 2026?
Yes for most analysts: it is the standard table library in Python, it reached version 3 in January 2026, and most AI assistants write pandas by default, so you need to read and check it.
Should I learn Python or SQL first for data analysis?
If your data lives in a database you query daily, SQL first. If it arrives as exports and spreadsheets, Python with pandas first. Most analysts end up using both.
Can I do data analysis in Python without installing anything?
Yes, with a hosted notebook such as Google Colab. Read its data terms first: Google's FAQ says AI features are on by default for eligible users and prompts and output may be reviewed and kept up to 18 months, so use sample data unless your organisation approves.
What is the difference between Jupyter and VS Code for notebooks?
Both run the same .ipynb notebooks with a Python kernel. JupyterLab is a browser-based environment; VS Code runs notebooks through its Jupyter extension alongside your scripts and tests.
How do I check code an AI assistant wrote?
Read every line, run it yourself from a fresh kernel, compare the result with an independent total, add a test or assert, check any new package name on pypi.org, and log what you checked.
Do I need statistics to be a data analyst?
You need the basics: medians and spread, rates from totals, and how to plan and report a simple test with an effect size and interval. Two-proportion tests, Welch's t-test and chi-square cover most business comparisons.
How do I measure my Python data analysis skills?
Practise on messy data with known answers, then take an executed-script assessment such as AssessAll's Data Analysis in Python Assessment, and turn the gaps into a development plan.