First-year courses · Computing

Intro data science: data frames, cleaning and models

Intro data science flashcards: tidy data, data frame operations, cleaning, missing values, summary statistics, charts, train and test splits and simple models.

40 cards5 topicsAbout 15 minutes a roundTypical first-year introduction to data science

Print cards to cut out (PDF)Print a study list (PDF)

About this deck

This deck is for students in a first-year data science course, or anyone learning to work with tables of data in a tool such as pandas, R or a spreadsheet. It covers how data should be organized (tidy data, variable types, data frames and indexes), the everyday operations of filtering, selecting, grouping, joining and reshaping, the work of cleaning data (missing values, duplicates, outliers, scaling and encoding), summary statistics, choosing the right chart, and the first ideas of machine learning: supervised and unsupervised learning, training and test sets, overfitting, regression, classification metrics and clustering.

The concepts are written to be language-neutral, with pandas names mentioned where they help. Basic statistics (mean, median, standard deviation) and comfort reading a table are enough to start; some programming experience makes the operations cards easier to picture.

The best way to study is to open a small dataset, for example a class survey or a public open-data file, and try each operation as you meet it. Predict the shape of the result (how many rows and columns) before running the command. For the modelling cards, focus on why each step exists, such as why the test set must stay untouched, rather than on formulas.

Common mistakes: filling missing values without asking why they are missing; computing scaling statistics on the whole dataset before splitting, which leaks test information into training; and reading a strong correlation as proof that one variable causes the other.

Before you start: High school statistics (mean, median, standard deviation, graphs) and, ideally, a little programming in Python or R.

All 40 cards

Cover the definitions with a sheet of paper and test yourself, or use the study modes above.

Data and structure

Tidy data: A layout where each variable is a column, each observation is a row, and each type of observational unit has its own table.Example: A table with columns student, course, grade, one row per student per course.
Categorical versus numerical variables: Categorical variables place observations in groups (province, colour); numerical variables are measured quantities on which arithmetic makes sense.Example: Postal code looks numeric in some countries but is categorical: averaging codes is meaningless.
Ordinal variable: A categorical variable whose categories have a natural order but uneven or unknown gaps between them.Example: Survey answers: disagree, neutral, agree.
Data frame: A two-dimensional table with labelled columns that may each hold a different data type, plus a row index.Example: pd.DataFrame({'city': ['Calgary', 'Halifax'], 'pop_k': [1306, 440]}) has 2 rows and 2 columns.
Series: A one-dimensional labelled array; each column of a pandas DataFrame is one.Example: df['pop_k'] returns a one-dimensional object of populations indexed like the rows.
Index: The row labels of a data frame, used to align data in operations and to look rows up.Example: Setting the row labels to student ID lets you fetch a row by ID instead of by position.
Data type: The storage type of a column, such as integer, float, string, boolean, datetime or category; it controls which operations work.Example: Ages read in as text '19', '20' must be converted to integers before averaging.

Data frame operations

Boolean filtering: Selecting rows with a true/false mask built from a condition on columns.Example: df[df['age'] >= 18] keeps only rows where age is at least 18.
Selecting columns: Picking a subset of columns, by name, to work with a narrower table.Example: df[['name', 'grade']] returns a two-column data frame.
loc versus iloc: One selects by labels and includes both ends of a label slice; the other selects by integer positions and excludes the end of a slice.Example: df.iloc[0:3] gives rows at positions 0, 1, 2; df.loc['a':'c'] includes row 'c'.
Group by (split-apply-combine): Split rows into groups by key, apply a summary to each group, and combine the results into a new table.Example: Grades 70 and 80 in section A, 90 in B: mean by section gives A = 75, B = 90.
Merge or join: Combining two tables on a shared key column. An inner join keeps keys found in both; a left join keeps every row of the left table.Example: Students joined to a fees table by ID; a student with no fee row gets a missing value under a left join.
Pivot table: Reshapes long data into a grid, with one variable as rows, one as columns and a summary of a third in the cells.Example: Rows = province, columns = year, cells = total enrolment.
Concatenation: Stacking tables with the same columns on top of each other, or side by side.Example: Combining January.csv and February.csv into one table of both months.
Vectorized operation: An operation applied to a whole column at once instead of in an explicit loop; usually much faster and shorter.Example: df['total'] = df['price'] * df['qty'] multiplies every row in one line.

Cleaning and preparation

Missing values: Entries with no recorded value, often shown as NaN or NA; most summaries skip them, but counts and models must handle them deliberately.Example: df.isna().sum() counts the empty entries in each column.
Dropping versus imputing: Gaps in data can be removed (drop rows or columns) or filled with an estimate such as the mean, median or a model prediction.Example: Values 4, 6, missing, 10: mean imputation fills 20/3 = 6.67; median imputation fills 6.
Duplicate records: Rows that repeat the same observation, often from merges or repeated data entry; they inflate counts and bias summaries.Example: df.drop_duplicates() removes exact repeats; matching on an ID column catches near-duplicates.
Outlier and the IQR rule: A value far from the rest. A common flag is any value below Q1 - 1.5 IQR or above Q3 + 1.5 IQR.Example: Q1 = 10, Q3 = 30, IQR = 20: values below -20 or above 60 are flagged.
Standardization (z-score): Rescaling a variable to mean 0 and standard deviation 1 with z = (x - mean)/sd.Example: Score 85 with mean 70 and sd 10: z = 1.5.
Min-max scaling: Rescaling values to the range 0 to 1 with (x - min)/(max - min).Example: Range 10 to 60: the value 30 becomes 20/50 = 0.4.
One-hot encoding: Turning a categorical column into several 0/1 columns, one per category, so models can use it.Example: colour in {red, blue, green}: 'blue' becomes red = 0, blue = 1, green = 0.
Data leakage: When information that would not be available at prediction time, such as test-set statistics or the target itself, slips into training.Example: Scaling with the mean of the full dataset before splitting leaks test information.

Summaries and charts

Mean versus median: The mean is the arithmetic average and is pulled by extreme values; the median is the middle value and resists them.Example: 1, 2, 3, 4, 100: mean = 110/5 = 22, median = 3.
Correlation coefficient: Pearson r measures the strength and direction of a linear relationship, from -1 to 1; 0 means no linear relationship.Example: x = 1, 2, 3 and y = 2, 4, 6 give r = 1.
Correlation versus causation: Two variables can move together because of a confounder, chance or reverse causation, so association alone does not prove cause.Example: Ice cream sales and drownings both rise in summer heat.
Histogram: A chart of a numerical variable's distribution using adjacent bins; bar heights are counts or densities.Example: Exam scores binned 50-59, 60-69, 70-79 show where most students landed.
Box plot: Shows the median, quartiles and whiskers of a numerical variable, with points beyond the fences drawn separately.Example: Side-by-side versions compare commute times across cities.
Scatter plot: Plots one numerical variable against another, one point per observation, to show relationships and clusters.Example: Hours studied against exam score.
Choosing a bar chart or line chart: Bars compare amounts across categories and should start at zero; lines show change over an ordered variable such as time.Example: Enrolment by faculty as bars; monthly enrolment over five years as a line.

Basic machine learning

Supervised learning: Learning a mapping from inputs to a known target using labelled examples; split into regression (numeric target) and classification (category).Example: Predicting house price from size and location.
Unsupervised learning: Finding structure, such as groups or patterns, in data that has no target labels.Example: Grouping customers by purchase habits without predefined segments.
Train and test split: Holding back part of the data to estimate how a model performs on new data; the model never sees the held-back part during training.Example: An 80/20 split of 1000 rows gives 800 for training and 200 for testing.
Overfitting: A model that memorizes noise in the training data: excellent training score, poor score on new data.Example: Training accuracy 99%, test accuracy 70% suggests it.
Cross-validation: Repeatedly splitting the training data into folds, training on all but one and validating on the remaining one, then averaging the scores.Example: 5-fold: five models, each validated on a different fifth of the data.
Simple linear regression: Fits y = b0 + b1 x by minimizing the sum of squared residuals; b1 is the predicted change in y per unit of x.Example: With y = 1 + 2x, x = 4 gives a prediction of 9.
Confusion matrix: A table of a classifier's true positives, false positives, true negatives and false negatives.Example: TP 40, FP 10, FN 20, TN 30: accuracy = 70/100 = 0.70.
Precision and recall: Precision = TP/(TP + FP), the share of predicted positives that are right; recall = TP/(TP + FN), the share of real positives found.Example: TP 40, FP 10, FN 20: precision 0.80, recall 40/60 = 0.67.
k-nearest neighbours: Predicts a new point's label from the majority (or average) of the k closest training points.Example: k = 3, neighbours labelled A, A, B: predict A.
k-means clustering: Partitions data into k clusters by repeatedly assigning points to the nearest centre and moving each centre to its cluster's mean.Example: Points 1, 2, 10, 11 with k = 2 settle on centres 1.5 and 10.5.