Background: The Detailed Results Processing Challenge

The module we chose for our first Polars implementation was our detailed results processor. This critical component takes full model results and produces detailed aggregations of expected hospital activity across inpatient (IP), outpatient (OP), and Accident & Emergency (A&E) services.

The processing involves:

1. Loading hundreds of results from Azure storage
2. Merging these with reference datasets
3. Processing 256 Monte Carlo simulation runs
4. Aggregating the results into statistical summaries
5. Validating the outputs against aggregated results
6. Saving the final results as both CSV and Parquet files

This workload is particularly demanding because:

• Each run processes thousands of records
• The full pipeline handles 256 separate runs (for statistical robustness)
• Memory consumption grows substantially during processing
• The reference datasets contain multiple categorical dimensions for analysis

Let’s look at how the implementations differ between our original Pandas version and the new Polars version.

Implementation Comparison

def _process_inpatient_results(
    ctx: ProcessContext,
    output_dir: str,
) -> None:
    # Function contains ~150 lines handling everything from:
    # - Data loading
    # - Reference preparation
    # - Results processing for all runs
    # - Dictionary aggregation
    # - Validation and saving
    # ...

def _process_inpatient_results(
    ctx: ProcessContext,
    output_dir: str,
) -> None:
    # Higher-level orchestration (~40 lines)
    # ...

def _load_ip_reference_data(ctx: ProcessContext) -> pl.DataFrame:
    # Focused on just loading reference data (~15 lines)
    # ...

def _process_ip_run(
    run: int,
    reference_df: pl.DataFrame,
    params: dict,
    model_runs: dict,
    ip_columns: list[str],
) -> None:
    # Processes a single run only (~30 lines)
    # ...

def _validate_ip_results(
    model_runs_df: pl.DataFrame,
    actual_results_df: pl.DataFrame
) -> None:
    # Focused validation logic (~20 lines)
    # ...

# Load and merge data
original_df = az.load_data_file(data_connection, model_version_data, trust, "ip", baseline_year)
reference_df = original_df.copy().drop(columns=["speldur", "classpat"])
# ...
merged = reference_df.merge(df, on="rn", how="inner")

# Load and join data
original_df = az_pl.load_data_file(data_connection, model_version_data, trust, "ip", baseline_year)
reference_df = original_df.drop(["speldur", "classpat"])
# ...
merged = reference_df.join(df, on="rn", how="inner")

# Fill missing values
original_df = az.load_data_file(...).fillna("unknown")

# Fill null values with more explicit control
original_df = az_pl.load_data_file(...)
original_df = original_df.with_columns(
    [
        pl.col(col).map_elements(
            lambda x: None if x == "" else x, return_dtype=pl.Utf8
        )
        for col in original_df.select(pl.col(pl.Utf8)).columns
    ]
).fill_null("unknown")

# Extract a specific value
detailed_beddays_principal = (
    model_runs_df.loc[
        (
            slice(None),
            slice(None),
            slice(None),
            slice(None),
            slice(None),
            slice(None),
            slice(None),
            "beddays",
        ),
        :,
    ]
    .sum()
    .loc["mean"]
    .astype(int)
)

# Extract the same value with explicit filtering
detailed_beddays_principal = int(
    model_runs_df.filter(pl.col("measure") == "beddays")
    .select(pl.col("mean").sum())
    .item()
)

# Each operation executes immediately
result = (
    df[df["measure"] == "beddays"]
    .groupby(["sitetret", "age_group"])
    .agg({"value": "sum"})
    .reset_index()
)

# Builds an execution plan that can be optimised
result = (
    df.lazy()
    .filter(pl.col("measure") == "beddays")
    .groupby(["sitetret", "age_group"])
    .agg(pl.col("value").sum())
    .collect()  # Only executes the plan when needed
)

# Choose batch size and load with caching
batch_size = 30  # Balance between memory usage and I/O performance
df = az.load_model_run_results_file(
    container_client=results_connection,
    params={
        # ... parameters
        "batch_size": batch_size,  # This enables batch loading
    },
)

# Clean up memory
del model_runs_df, model_runs, original_df, reference_df
gc.collect()
logger.info(f"Memory cleaned after IP processing, current usage: {get_memory_usage():.2f} MB")

# Similar batch loading pattern
batch_size = 50  # Note larger batch size than Pandas
df = az_pl.load_model_run_results_file(
    container_client=results_connection,
    params={
        # ... parameters
        "batch_size": batch_size,
    },
)

# Similar memory clean-up
def _clean_ip_memory(
    model_runs_df: pl.DataFrame,
    model_runs: dict,
    original_df: pl.DataFrame,
    reference_df: pl.DataFrame,
) -> None:
    del model_runs_df, model_runs, original_df, reference_df
    # ... cache clearing and collection
    gc.collect()
    logger.debug(f"Memory cleaned after IP processing, current usage: {get_memory_usage():.2f} MB")

# Count rows with empty sitetret for validation
empty_sitetret_count = results.filter(
    pl.col("sitetret").is_null() | (pl.col("sitetret") == "")
).height
if empty_sitetret_count > 0:
    logger.debug(f"Found {empty_sitetret_count} rows with empty sitetret in run {run}")

# Ensure consistent numerical representation for statistics
op_model_runs_df = op_model_runs_df.with_columns(
    [
        pl.col("lwr_ci").fill_nan(0.0),
        pl.col("median").fill_nan(0.0),
        pl.col("mean").fill_nan(0.0),
        pl.col("upr_ci").fill_nan(0.0),
    ]
)

Performance Benefits

The most compelling reason to switch from Pandas to Polars is its columnar storage, expression-based API, and query optimisation engine. In our benchmarks, we measured total execution time including I/O operations (reading from Azure Blob Storage and writing to local files), providing a realistic picture of performance in production environments. Our experience reimplementing our Pandas code to Polars taught us that:

1. Processing Speed: Our Polars implementation, without any additional optimisations that might increase performance, is more than 32% faster than its Pandas counterpart. Our benchmark included I/O time in speed calculations, which relies on Azure and network speed as well as computational performance of the specific machine used to perform this benchmark. Therefore, with the benefit of hindsight, introducing better separation of concerns such as benchmarking dataframe operations separately from loading data from Azure Blob Storage would likely yield even better performance gains. The Polars team’s comprehensive benchmarks confirm its improved performance across a wide range of operations.

2. I/O Performance: The Polars implementation handles file operations more efficiently, particularly when loading and saving Parquet files, as it’s designed to work natively with the Apache Arrow format.

3. Parallel Processing: Polars automatically utilises multiple CPU cores by default, whereas Pandas operations are typically single-threaded. While we haven’t explicitly configured parallel execution in our implementation, we benefit from this feature automatically.

4. Larger-than-RAM Processing: Polars can handle datasets larger than available RAM through its lazy API and streaming engine. This is crucial for our future work as healthcare datasets continue to grow in size and complexity.

5. Lightweight Dependencies: Unlike Pandas, which depends on NumPy and several other libraries, Polars is certainly more lightweight with no Python dependencies. This makes deployment simpler and reduces potential version conflicts in complex projects.

For context, processing a typical scenario with 256 runs that previously took about 57 minutes with Pandas now completes in 38 minutes with Polars, with basic Polars API knowledge.

Trade-offs and Considerations

Despite the clear performance advantages, there are some trade-offs to consider:

Practical Advice for Adoption

Based on our experience, here are some recommendations for teams considering a similar transition:

Conclusion

Our experiment, transforming our detailed results processor from Pandas-based to Polars has been didactic and positive. The expression-based API is more readable, making our code more maintainable. As a bonus, our pipeline can run faster -especially when processing larger datasets.

For teams working with large datasets, particularly those experiencing performance bottlenecks with Pandas, Polars represents a compelling alternative that maintains much of the familiar dataframe paradigm while delivering measurable performance improvements.

We’re continuing to investigate ways to improve our codebase, its speed of execution, memory and overall efficiency, and looking forward to leveraging more effective approaches that’ll render our code even more powerful.

This blog post reflects the experience of the Strategy Unit’s Data Science team in implementing Polars for detailed results processing. The performance improvements mentioned are specific to our workloads and may vary depending on your specific use case.

Can you sign your own commits?

Absolutely — and you probably should!
This isn’t just a GitHub feature; it’s built directly into Git.

Below is a screenshot of our nhp_model, where all the commits I’ve authored have been signed with my GPG key.

Setting up GPG can be a bit of a faff, but there’s an easier way: using SSH keys to sign commits.
You may already have SSH set up for pushing and pulling from GitHub, so this is a no-brainer.
Even if you don’t, setting up SSH is quick.

Easy approach: using SSH

For more details, see “Sign commits with SSH keys” from GitLab’s docs, but here’s the short version:

First, generate an SSH key if you haven’t already. Open a terminal (on Windows you might need Git Bash) and run:

ssh-keygen -t ed25519 -C "my.email@example.com"

This creates a new file called ~/.ssh/id_25519.pub, where ~ is your home directory (for example, on Windows: C:\Users\thomas.jemmett\.ssh\id_25519.pub).

Then run:

git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_25519.pub
git config --global tag.gpgsign true
git config --global commit.gpgsign true

From now on, your commits will be signed — but GitHub won’t yet recognise the signature! You need to upload your SSH key for signing: see“Adding a new SSH key to your account”, GitHub Docs.

You need to make sure that you set this as a key for signing, not for authentication. If you already have uploaded your key for authentication, you need to upload it again for signing. Using the same key for both is fine.

Harder approach: GPG

While GitHub supports SSH-signed commits, not all providers do. GPG is a more established way of performing these cryptographic operations.

It does take more setup, and you may need to install additional tools. Rather than going step by step here, you should read GitHub’s docs on creating a key.

If you do create a GPG key, consider using a modern algorithm like ECDSA or EdDSA.

It’s also a good idea to send your new key to a few keyservers (e.g. pgp.mit.edu, keyserver.ubuntu.com, keys.openpgp.org).

If some of your colleagues also use GPG, you might sign their keys to help establish a Chain of trust.

Paranoid approach

Once your commits are verifiable, you need to ensure your private key never leaks. If an attacker obtained your SSH or GPG key, they could create commits pretending to be you and sign them so others would completely trust them.

If you’re ultra-paranoid (or a maintainer of a popular repository), consider this:

Generate a GPG master key on an air-gapped machine (i.e. not connected to the internet), use it to generate short-lived subkeys, then copy the subkeys to a hardware security key (HSK) like a Yubikey.

If you want a good guide to follow, you might want to read this series of blog posts: (though, I would choose ECC over RSA as suggested). Part 6 explains how to get this to work with a Yubikey.

A good guide is the blog series Applying GPG and Yubikey (I’d recommend ECC over RSA as suggested). Part 6 covers integrating this with a YubiKey.

This approach stores your cryptographic secrets in a way they can’t be extracted (HSKs only import keys, not export them), meaning an attacker would have to physically steal your device.

Curating the product backlog

This is probably my favourite thing about Positron. The GitHub interface in RStudio was ok, but I still found myself using the command line, which felt like it interrupted my flow of work to a degree. Positron provides for the ability to never have to leave the IDE. Once you have authenticated your GitHub account in Positron, you can click the GitHub icon in the activity panel to see all the pull requests and issues for that repo. You can create PRs from that menu too – it’s all self-explanatory and works really well. Clicking on one of your repo issues will also open it as an additional tab in the editor pane, and you can interact with it there just as you would at github.com! No need to have browsers and shells open, it’s all just there. This helps my workflow massively, allowing me to keep my train of thought going.

Writing plans and processes and sharing them

Quarto, which is already bilingual, is the tool I use to write meeting notes, lists and plans, all within a locally version-controlled Quarto book. The whole book is searchable, so it’s easy to find the notes I’m looking for, and Positron makes working with Quarto very easy. Positron comes with a bootstrapped Quarto extension and the CLI is bundled with Positron, so no extra setup is needed. I can create my planning documents, with nice mermaid flow diagrams and such, and I can publish them easily from Positron to share with my team. I’ve written a package that I can use within the IDE to automatically append notes to any of the .qmd pages of my notes book. I can even create diagrams and charts in the IDE, and save them into my notes. If the diagram is complex, I can append the code chunks and output to my book too, since that’s what Quarto is great at.

I love Positron’s feature that allows scrolling back through the iterations of a plot too, which makes rolling the code back to a preferred solution nice and simple.

Testing new functionality and UX

I need to understand how our products are experienced by our users, so it helps me to run our apps in development locally. Using Positron allows me to use a very similar workflow to my familiar RStudio one, including using {devtools} for documenting and running package checks and using devtools::load_all() to test out new functionality locally. In addition to this, Positron makes working with tests really simple (which is saying something coming from me!). The ‘testing’ icon in the activity pane opens a dedicated interface sidebar, which neatly displays your tests and their results (provided you are in a package folder). It lets you easily rerun failing tests, see their error messages, and rerun batches of tests or all tests at once. This is a really great addition to the IDE.

Juggling many projects

I flit between projects often. Double-clicking the R Project file allows you to start a new RStudio session with the project already loaded. But there is no direct equivalent of the .Rproj file for Positron.
This is a little jarring at first, but opening a folder to begin working in a project is a familiar workflow with VSCode.
I wanted a quick and easy way to switch between projects, and have found the Project Manager Extension useful for this. It creates a new multi-folder icon in the activity pane, which when clicked gives a neat overview of saved projects (folders), which I can tag easily to bundle together related projects. Lovely!

Helpers for doing good data science

I don’t want to lose my skills as a data scientist; I like to dabble and make useful tools for myself, and very occasionally I get my hands dirty writing code with the team. Positron is helping me write better code, quickly, which helps me look like a data scientist (I hope!). My favourite features so far are having my code formatted automatically when I save an R or Quarto file by the excellent Air formatter (shipped as standard with Positron), using code snippets to get the syntax of common functions right first time, and the cool multi-cursor for editing in multiple places at once!

Overall, Positron feels like a nice place to be. I’m finding it to be an IDE that helps with so many aspects of being a product owner, and that can only help us to keep building the right thing at the right time.

What’s the problem with missing data?

When datasets contain gaps, whether due to non-response in surveys, data entry errors, or system malfunctions, the integrity of the analysis is compromised.

In my case, missing data presented two main challenges. First, the missingness prevented some of my intervention sites from being matched with control sites, which reduced the overall dataset size and excluded important information from my analysis. Second, the type of missingness in my data meant that excluding incomplete records introduced biases into the dataset.

Understanding the type of missingness in your dataset is essential for choosing the right approach to handle it.

Handling missingness

Handling missingness can be approached through various techniques, each with its own strengths and weaknesses.

One common method is to delete rows containing missing data (i.e. complete-case analysis). This method is simple to do and is often a default method for statistical packages, however, it is only appropriate for situations where the data is missing completely at random (MCAR).

Another approach is imputation, where missing values are filled in based on observed data. Simple methods include the mean or median imputation, which replaces missing values with the average of the observed values. While straightforward, these methods can reduce variability in the dataset. More sophisticated techniques, such as regression imputation, use regression models to predict and fill in missing values based on other variables, providing potentially more accurate estimates, but introducing bias if the model is not well-specified.

For more advanced handling of missing data, multiple imputation creates several different imputed datasets and combines the results to account for uncertainty in the missing data. This approach offers more robust estimates and better reflects the variability in the data.

Applied example

Now that we’ve covered the theory of missingness, its impact on analyses, and ways to deal with it, let’s put these techniques into practice using a fictional dataset.

• We’ll create a sample dataset that simulates authentic relationships between age, gender and salary. This will provide the foundation for our experiment.

• Next, we’ll intentionally introduce non-random missingness into the dataset to create a realistic scenario with incomplete data.

• We’ll then fill in the missing data using four different imputation techniques.

• Finally, we’ll compare the results from each imputation technique with the original dataset to evaluate their performance and see how well they work with this example dataset.

# decide the number of rows to create
rows <- 1000

# Ensure reproducibility
set.seed(123)

# Generate the data
df_complete <-
  tibble::tibble(
    # age and gender are independent
    age = rpois(n = rows, lambda = 40),
    gender = sample(x = c('Male', 'Female'), size = rows, replace = TRUE),

    # generate salary based on age group
    salary_age = dplyr::case_when(
      age %in% 18:21 ~ rnorm(n = rows, mean = 24000, sd = 5000),
      age %in% 22:29 ~ rnorm(n = rows, mean = 33000, sd = 6000),
      age %in% 30:39 ~ rnorm(n = rows, mean = 40000, sd = 7000),
      age %in% 40:49 ~ rnorm(n = rows, mean = 43000, sd = 8000),
      age %in% 50:59 ~ rnorm(n = rows, mean = 41000, sd = 7000),
      age > 59       ~ rnorm(n = rows, mean = 36000, sd = 5000)
    ),

    # gender pay gap of 7% across all age ranges
    salary = dplyr::case_match(
      gender,
      'Female' ~ salary_age * 0.93,
      .default = salary_age
    ) |>
      round(digits = 0)
  ) |>
  dplyr::select(-salary_age)

Warning: `case_match()` was deprecated in dplyr 1.2.0.
ℹ Please use `recode_values()` instead.

# see a sample
df_complete |> 
  dplyr::slice_head(n = 10) |> 
  gt::gt() |> 
  gt::opt_stylize(style = 5, add_row_striping = FALSE)

#' Function to create a density plot
#' 
#' @param df Tibble of data
#' @param x_val Variable to plot on the x axis
#' @param x_label String - Name of the variable in the x axis
#' @param summary String - Summary stats (mean and sd) for `x_val`
#' @param complete Boolean (default = FALSE) - Is this complete data?
plot_density <- function(df, x_val, x_label, summary, complete = FALSE) {
  
  # define our fill colour:
  # reference (complete) data is coloured orange,
  # comparison data is coloured blue
  if (complete) {
    fill_colour <- adjustcolor(
      col = "#f9bf07",
      alpha.f = 0.2
    )
  } else {
    fill_colour <- adjustcolor(
      col = "#5881c1",
      alpha.f = 0.2
    )
  }
  
  # produce the plot
  p <- 
    df |> 
    ggplot2::ggplot(ggplot2::aes(x = {{x_val}})) +
    ggplot2::geom_density(
      fill = fill_colour,
      outline.type = "upper"
    ) +
    ggplot2::labs(title = x_label) +
    ggplot2::annotate(
      geom = 'text',
      label = summary,
      size = 14/ggplot2::.pt,
      x = mean(df |> dplyr::pull({{x_val}}), na.rm = TRUE),
      y = 0,
      hjust = 0.5,
      vjust = 0
    ) +
    ggplot2::theme_minimal(base_size = 18) +
    ggplot2::theme(
      # titles
      plot.title = ggplot2::element_text(size = 22),
      # axes
      axis.ticks = ggplot2::element_blank(),
      axis.line = ggplot2::element_blank(),
      axis.title = ggplot2::element_blank(),
      axis.text.y = ggplot2::element_blank(),
      axis.text.x = ggplot2::element_text(size = 18),
      # grid lines
      panel.grid = ggplot2::element_blank(),
      panel.border = ggplot2::element_blank(),
      # background
      plot.background = ggplot2::element_blank()
    )
  
  # add thousands suffix to salary outputs
  if (x_label == "Salary") {
    p <-
      p +
      ggplot2::scale_x_continuous(
        labels = scales::label_number(suffix = 'k', scale = 1e-3)
      )
  }
  
  # return the plot
  return(p)
}


#' Function to compare density distributions
#'
#' @param df_complete Tibble - the 'complete' dataset
#' @param df Tibble - the comparison dataset containing missing or imputed values
#'
#' @returns ggplot2 object
compare_distributions <- function(df_complete, df = NA) {
  
  # summary stats for age and salary
  complete_summary_age = glue::glue(
    "Mean: {round(mean(df_complete$age, na.rm = TRUE), digits = 2)}\n",
    "SD: {round(sd(df_complete$age, na.rm = TRUE), digits = 2)}"
  )
  
  complete_summary_salary = glue::glue(
    "Mean: {round(mean(df_complete$salary, na.rm = TRUE)/1000, digits = 2)}k\n",
    "SD: {round(sd(df_complete$salary, na.rm = TRUE)/1000, digits = 2)}k"
  )
  
   # get the 'complete' density plots
  complete_age <-
    plot_density(
      df = df_complete,
      x_val = age,
      x_label = "Age",
      summary = complete_summary_age,
      complete = TRUE
    )
  
  complete_salary <-
    plot_density(
      df = df_complete,
      x_val = salary,
      x_label = "Salary",
      summary = complete_summary_salary,
      complete = TRUE
    )
  
  # check whether a comparison dataset has been provided
  if (!missing(df)) {
    
    # summary stats for age and salary
    input_summary_age = glue::glue(
      "Mean: {round(mean(df$age, na.rm = TRUE), digits = 2)}\n",
      "SD: {round(sd(df$age, na.rm = TRUE), digits = 2)}"
    )
  
    input_summary_salary = glue::glue(
      "Mean: {round(mean(df$salary, na.rm = TRUE)/1000, digits = 2)}k\n",
      "SD: {round(sd(df$salary, na.rm = TRUE)/1000, digits = 2)}k"
    )
    
    # get the comparison density plots
    comparison_age <-
      plot_density(
        df = df,
        x_val = age,
        x_label = "Age",
        summary = input_summary_age,
        complete = FALSE
      )
    
    comparison_salary <-
      plot_density(
        df = df,
        x_val = salary,
        x_label = "Salary",
        summary = input_summary_salary,
        complete = FALSE
      )
    
    # combine the charts to compare
    plots <- 
      patchwork::wrap_plots(
        complete_age, comparison_age,
        complete_salary, comparison_salary,
        nrow = 2
      )
  } else {
    # there is only the complete dataset included, so show that
    plots <-
      patchwork::wrap_plots(
        complete_age, complete_salary,
        ncol = 1
      )
  }
  
  # return the plots
  return(plots)
  
}

compare_distributions(df_complete = df_complete)

# take a copy of the complete data
df_missing <- df_complete
seed <- 100

# age - where salary > 45000
set.seed(seed) # for reproducibility
df_missing$age[
  sample(which(df_complete$salary > 45000), 
         size = 100, 
         replace = FALSE)] <- NA
  
# salary - 90 where age > 45, 10 from rest
set.seed(seed) # for reproducibility
df_missing$salary[
  sample(which(df_complete$age < 40), 
         size = 100, 
         replace = FALSE)] <- NA

# gender - 100 randomly
set.seed(seed) # for reproducibility
df_missing$gender[
  sample(1:rows, 
         size = 100, 
         replace = FALSE)] <- NA

# see a sample
missing_cell_style <- gt::cell_fill(color = "#ec6555")
df_missing |> 
  dplyr::slice_head(n = 10) |> 
  gt::gt() |> 
  gt::opt_stylize(style = 5, add_row_striping = FALSE) |> 
  # highlight cells with missing values
  gt::tab_style(
    style = missing_cell_style,
    locations = gt::cells_body(columns = age, rows = is.na(age))
  ) |> 
  gt::tab_style(
    style = missing_cell_style,
    locations = gt::cells_body(columns = gender, rows = is.na(gender))
  ) |> 
  gt::tab_style(
    style = missing_cell_style,
    locations = gt::cells_body(columns = salary, rows = is.na(salary))
  )

# remove records containing missing values
df_removal <-
  df_missing |> 
  na.omit()

# see the result
compare_distributions(df_complete = df_complete, df = df_removal)

df_average <- 
  df_missing |> 
  dplyr::mutate(
    age = dplyr::case_match(
      age,
      NA ~ mean(age, na.rm = TRUE),
      .default = age
    ) |> 
      round(digits = 0),
    
    salary = dplyr::case_match(
      salary,
      NA ~ mean(salary, na.rm = TRUE),
      .default = salary
    ) |> 
      round(digits = 0)
  )

# see the result
compare_distributions(df_complete = df_complete, df = df_average)

# model age and salary
model_age <- glm(
  data = df_missing |> na.omit(),
  formula = age ~ salary,
  na.action = "na.exclude"
)

model_salary <- glm(
  data = df_missing |> na.omit(),
  formula = salary ~ age,
  na.action = "na.exclude"
)

# fill in details using regression
df_regression <-
  df_missing |> 
  dplyr::mutate(
    age_pred = predict(
      object = model_age,
      newdata = data.frame(
        gender = df_average$gender, 
        salary = df_average$salary)
    ) |> 
      round(digits = 0),
    
    salary_pred = predict(
      object = model_salary,
      newdata = data.frame(
        gender = df_average$gender, 
        age = df_average$age)
    ) |> 
      round(digits = 0),
    
    age = dplyr::coalesce(age, age_pred),
    salary = dplyr::coalesce(salary, salary_pred)
  ) |> 
  dplyr::select(-c(age_pred, salary_pred))

# see the result
compare_distributions(df_complete = df_complete, df = df_regression)

# let {mice} suggest an imputation method for each variable
init <- mice::mice(df_missing, maxit = 0)

# calculate the imputed values
imputed <- mice::mice(
  data = df_missing,          # data to be used
  m = 10,                     # number of multiple imputations (dflt = 5)
  method = init$method,       # matching method - will go with default
  seed = 123,                 # for reproducibility
  maxit = 10,                 # 10 iterations (default = 5)
  printFlag = FALSE           # don't print history to the console
)

# complete the data
df_mice <- mice::complete(imputed)

# see the result
compare_distributions(df_complete = df_complete, df = df_mice)

# list the data showing different imputation approaches
data <- 
  tibble::tribble(
    ~df, ~df_name,
    df_complete, "Original",
    df_removal, "Complete-case",
    df_average, "Average",
    df_regression, "Regression",
    df_mice, "MICE"
  )

# summarise the results to a single df
df_summary_test <-
  purrr::map2_dfr(
    .x = data$df,
    .y = data$df_name,
    .f = \(.x, .y) {
      .x |> 
        dplyr::summarise(
          set = .y,
          age_mean = mean(age),
          age_sd = sd(age),
          salary_mean = mean(salary),
          salary_sd = sd(salary)
        )
    }
  ) |> 
  # split age and salary to separate rows
  tidyr::pivot_longer(
    cols = -set,
    names_to = c("measure", ".value"),
    names_pattern = "(.*)_(.*)"
  ) |> 
  # convert salary to thousands to put on the same scale as age
  dplyr::mutate(
    mean = dplyr::case_when(
      measure == "salary" ~ mean * 1e-3,
      .default = mean
    ),
    sd = dplyr::case_when(
      measure == "salary" ~ sd * 1e-3,
      .default = sd
    )
  )

# define a function to display results in a formatted table
#' Produce a summary table of results for a given measure
#'
#' @param df Tibble containing summary results from each imputation approach
#' @param .measure String identifying the measure to summarise (either 'age' or 'salary')
#'
#' @returns gt table
imputation_summary_table <- function(df, .measure = c("age", "salary")) {
  
  # get the original data as reference
  df_original <-
    df |> 
    dplyr::filter(measure == .measure, set == "Original") |> 
    dplyr::select(mean, sd)
  
  # filter the data for the specified measure and work out differences
  df_summary <-
    df |> 
    dplyr::filter(measure == .measure) |> 
    dplyr::mutate(
      # add the 'original' mean and sd values to each row
      mean_original = df_original$mean,
      sd_original = df_original$sd,
      
      # work out  differences between the imputation and original
      mean_difference = dplyr::case_when(
        set == "Original" ~ NA, # leave original blank
        .default = abs(mean - mean_original)
      ),
      sd_difference = dplyr::case_when(
        set == "Original" ~ NA, # leave original blank
        .default = abs(sd - sd_original)
      ),
      
      # determine which approach gives the overall lowest difference
      overall_difference = mean_difference + sd_difference,
      trophy = dplyr::case_when(
        overall_difference == min(overall_difference, na.rm = TRUE) ~ "🏆"
      )
    )
  
  # display as a formatted table
  tab <- 
    df_summary |> 
    gt::gt() |> 
    gt::tab_options(quarto.disable_processing = TRUE) |> 
    gt::fmt_number(decimals = 2) |> 
    gt::cols_hide(columns = c(
      "measure", 
      "mean_original", 
      "sd_original"
    )) |> 
    gt::sub_missing(missing_text = "") |> 
    gt::data_color(
      columns = c("mean_difference", "sd_difference", "overall_difference"),
      palette = c("#5881c1", "#dff9fb"),
      na_color = "white"
    ) |> 
    gt::cols_label(
      set = "Set",
      mean = "Mean",
      sd = "SD",
      mean_difference = "Mean",
      sd_difference = "SD",
      overall_difference = "Overall",
      trophy = "Trophy"
    ) |> 
    gt::tab_spanner(
      columns = c(mean_difference, sd_difference, overall_difference),
      label = "Difference from 'Original'"
    ) |> 
    gt::cols_align(columns = trophy, align = "center")
  
  return(tab)
}

# summarise the approaches for 'age'
imputation_summary_table(df = df_summary_test, .measure = "age")

# summarise the approaches for 'salary'
imputation_summary_table(df = df_summary_test, .measure = "salary")

Summary

Missing data can significantly impact the integrity of your findings, leading to skewed results and misleading conclusions. Understanding how to manage missing values is essential for drawing accurate insights.

We identified three types of missingness to consider:

Missing Completely At Random (MCAR): missingness is random and unrelated to observed or unobserved data.

Missing At Random (MAR): missingness is related to observed data, but not the missing data itself.

Missing Not At Random (MNAR): missingness is related to the unobserved data itself.

To explore techniques for handling missingness we created a synthetic dataset of 1,000 individuals with age, gender and salary, using authentic relationships. We introduced MAR missingness, simulating everyday scenarios.

We used four methods for handling missing data:

• Complete-case analysis

• Mean imputation

• Linear regression

• Multiple imputation using the MICE algorithm

The MICE algorithm emerged as the most accurate, closely mirroring the original dataset’s characteristics. Mean imputation and linear regression provided decent average estimates but distorted variability, leading to altered distributions.

This example serves as a powerful reminder of the importance of choosing the right approach for missing data. So, the next time you encounter missing values, remember: how you handle them can shape the story your data tells.

Practical tasks

Knowledge and skills required

Sharing the load

While there are benefits to maintaining the same person as scrummaster, we’ve switched recently to rotating the role among team members. This will provide respite for everyone and also improve big-picture knowledge across the team. In turn, this will reduce bus factor so that we won’t depend on any one individual to keep all the scrummaster knowledge in their brain.

Setup: Defining the problem

The key to success when you don’t have much time is to define your problem and objectives well. This helps keep your session focused and realistic. I had a good preparatory meeting with Andriana, our collaborator in the Evaluation team, who provided some examples of the free text data generated from Evaluation studies and their intended uses. We decided to use some data from a recent survey, which needed to be categorised into one or more of six different categories - a multilabel classification problem, with a small dataset of only 460 rows.

Our mission was to develop a model to help the Evaluation team categorise responses quickly, with a reasonable degree of performance.

The Hackathon format

We set aside one working day for our Hackathon. Having the event in our calendars was useful - we were able to focus uninterrupted on one problem all day, without other meetings and obligations. We kept our schedule pretty loose, but ended up with three meetings; one in the morning, one at lunchtime, and one at the end of the day. We opted to work separately, although we discussed the option of pair programming all day as well.

The meeting in the morning was mostly spent talking through the problem, and discussing the approaches we were going to take. This was so that we could avoid duplicating our efforts, and explore different potential solutions.

Our afternoon meeting was just a quick status update; I hadn’t done much coding by lunchtime because I’d spent hours trying to sort out my virtual environment! There’s quite a lot of setup required for some of these more state-of-the-art packages, and Windows isn’t the ideal operating system…

We reconvened in the evening to discuss what we’d achieved. It was great having quite a tight deadline - it made me really focus on the task at hand.

As an aside, I used ChatGPT to scaffold a lot of my code, and to help with debugging - it saved me so much time! I was able to use my time thinking through the problem and working out my approach, instead of trawling through Stackoverflow to figure out why my code wasn’t working. For me, the Hackathon would have been much less successful without it.

Between the three of us, we tried two different methods out.

Zero shot classification with fine-tuning

This method is more like traditional machine learning. It is relatively quick and requires less computing power than the zero shot prompting method. We used the huggingface facebook/bart-large-mnli model model, which could be finetuned as part of a human-in-the-loop training cycle. However, this method is quite slow, taking about 20 minutes to retrain the model even with such a small dataset.

flowchart TB
    A("Model labels data") --> B("Humans check and correct data")
    B --> n3("Model retrains")
    n3 --> A
    nn("Give model labels and data") --> A

    style A stroke:#000000,fill:#276DC2,color:#fff
    style nn stroke: #000000,fill:#ffde57
    style B stroke: #000000,fill:#ffde57
    style n3 stroke: #000000,fill:#276DC2,color:#fff

Zero shot prompting

This method utilises Ollama to run complex Large Language Models (LLMs) locally. This method relies on good prompt engineering, meaning that we had to test several different ways of asking the model to complete the task. No retraining was required. We just used our own machines to do this, and a computer with decent hardware can generate just over 80 tokens per second - meaning that it’s reasonably quick. Performance was slightly better, I would say, and it would even be possible to evaluate results automatically, with human oversight, using a method called “LLM eval”.

Next steps

Both methods performed reasonably well, and Andriana could see how they would be able to help the Evaluation team in the future. The Evaluation team would need explore how exactly these models might fit into the qualitative analysis process and which types of projects they would be relevant for, as these methods are likely more applicable to projects with ‘simpler’ data. We also spotted some opportunities for quick wins with better cleaning of the data. Having built these simple proofs of concept on our local machines, we now need to think about how we would productionise these approaches in the real world. I also want to spend time thinking about how the Data Science team and the Evaluation team can work more closely together in the future.

Overall, I would say that the Hackathon format was a great way to explore new methods in a collaborative way, without too much disruption of regular day to day responsibilities.

GitHub Apps

A GitHub App is a type of integration that you can build to interact with and extend the functionality of GitHub. You can build a GitHub App to provide flexibility and reduce friction in your processes, without needing to sign in a user or create a service account.

Common use cases for GitHub Apps include:

• Automating tasks or background processes

…

About GitHub Apps.

Sounds ideal, right? And it turns out it’s pretty easy to create your own app too! Well, there are a few steps, and a bit of boilerplate code to write, but I’ll get to that later.

If you explore that link, you’ll find all the details needed to create your own app—but I’ll quickly note the steps I took below.

#' Get GitHub JWT for an Application
#'
#' @param key Path to the private key file or a string containing the private
#'     key. Defaults to the environment variable `GITHUB_APP_PRIVATE_KEY`.
#' @param app_id GitHub App ID. Defaults to the environment variable
#'     `GITHUB_APP_ID`.
#' @param expiry_time Expiry time for the JWT in seconds. Defaults to 30s.
#'
#' @return A JSON Web Token (JWT) for the GitHub App.
get_github_jwt <- function(
    key = Sys.getenv("GITHUB_APP_PRIVATE_KEY"),
    app_id = Sys.getenv("GITHUB_APP_ID"),
    expiry_time = 30) {
  private_key <- openssl::read_key(key)

  now <- as.numeric(Sys.time())
  claim <- httr2::jwt_claim(
    iat = now,
    exp = now + expiry_time,
    iss = app_id
  )

  httr2::jwt_encode_sig(claim, key = private_key)
}

#' Get GitHub PAT from Installation Access Token
#'
#' @param jwt JSON Web Token (JWT) for the GitHub App. Defaults to the output of
#'     `get_github_jwt()`.
#' @param installation_id GitHub Installation ID. Defaults to the environment
#'     variable
#' @param github_api_ep The base URL for the GitHub API. Defaults to
#'     "https://api.github.com/".
#'
#' @return A personal access token (PAT) with permissions granted to the app.
get_github_app_installation_id <- function(
    jwt = get_github_jwt(),
    github_api_ep = "https://api.github.com/") {
  resp <- httr2::request(github_api_ep) |>
    httr2::req_url_path_append(
      "app",
      "installations"
    ) |>
    httr2::req_method("GET") |>
    httr2::req_auth_bearer_token(get_github_jwt()) |>
    httr2::req_headers(
      Accept = "application/vnd.github+json"
    ) |>
    httr2::req_perform()

  httr2::resp_check_status(resp)

  httr2::resp_body_json(resp)[[1]][["id"]]
}

#' Get GitHub PAT from Installation Access Token
#'
#' @param jwt JSON Web Token (JWT) for the GitHub App. Defaults to the output of
#'     `get_github_jwt()`.
#' @param installation_id GitHub Installation ID. Defaults to the output of
#'     get_github_app_installation_id()`.
#' @param github_api_ep The base URL for the GitHub API. Defaults to
#'     "https://api.github.com/".
#'
#' @return A personal access token (PAT) with permissions granted to the app.
get_github_iat_pat <- function(
    jwt = get_github_jwt(),
    installation_id = get_github_app_installation_id(),
    github_api_ep = "https://api.github.com/") {
  resp <- httr2::request(github_api_ep) |>
    httr2::req_url_path_append(
      "app",
      "installations",
      installation_id,
      "access_tokens"
    ) |>
    httr2::req_auth_bearer_token(jwt) |>
    httr2::req_headers(
      Accept = "application/vnd.github+json"
    ) |>
    httr2::req_method("POST") |>
    httr2::req_perform()

  httr2::resp_check_status(resp)

  httr2::resp_body_json(resp) |>
    purrr::pluck("token")
}

Putting it all together

Now that we can generate a token using our app, we can write a function to query the list of repositories.

We need to keep in mind that the API returns a maximum of 100 items per page. Fortunately, {httr2} makes it easy to iterate through paginated responses.

#' Get GitHub Repositories for an organisation
#'
#' @param org The name of the GitHub organisation.
#' @param pat Personal Access Token (PAT) for authentication. Defaults to the
#'     output of `get_github_iat_pat()`.
#' @param github_api_ep The base URL for the GitHub API. Defaults to
#'     "https://api.github.com/".
get_repos <- function(
    org,
    pat = get_github_iat_pat(),
    github_api_ep = "https://api.github.com/") {
  req <- httr2::request(github_api_ep) |>
    # build the url up, this should create something like
    # https://api.github.com/orgs/YOUR_ORG/repos
    httr2::req_url_path_append(
      "orgs",
      org,
      "repos"
    ) |>
    # add the correct requeste header for authentication using our PAT
    httr2::req_auth_bearer_token(pat) |>
    # additional headers GitHub expects to be passed to their API
    httr2::req_headers(
      Accept = "application/vnd.github+json",
      "X-GitHub-Api-Version" = "2022-11-28"
    ) |>
    # append url query parameters, this should look something like
    # https://api.github.com/orgs/YOUR_ORG/repos?per_page=100&page=1&sort=created
    httr2::req_url_query(
      per_page = 100, # anything between 1 and 100 max, as per the docs
      page = 1,
      sort = "created"
    )

  # because the API will only return a maximum of 100 items at a time, we need
  # to query multiple times for each page of results.
  # {httr2} makes this super easy, as the GitHub api returns page links in the
  # Link header as per RFC8288  (https://datatracker.ietf.org/doc/html/rfc8288)
  resps <- httr2::req_perform_iterative(
    req,
    next_req = httr2::iterate_with_link_url(rel = "next")
  )

  # ensure that we got a non-error response for each request
  purrr::walk(resps, httr2::resp_check_status)

  # get the data from each response, iterate over them and just extract the
  # "name" field that is returned for each item
  resps |>
    httr2::resps_data(httr2::resp_body_json) |>
    purrr::map_chr("name")
}

Finally, run the function like this:

# replace the below as required
Sys.setenv("GITHUB_APP_ID" = "[app_id]")
Sys.setenv("GITHUB_APP_PRIVATE_KEY" = "path-to-your.private-key.pem")

get_repos("Your Organisation")

KEY LEARNING

When working with text data, computers need to convert the words into numbers first before being able to work with them. Hence vectorisation - the process of converting numbers into words. There are a few different approaches and concepts which we’ll explore today

• Tokenization
• Bag of words
• TF-IDF
• n-grams
• Word2Vec embeddings

Finally we’ll look at (at a very very high level!) how Transformer/attention-based approaches to word vectorisation have transformed NLP

Tokenization

Breaking up texts into their individual components, or tokens

from nltk.tokenize import word_tokenize

text = "Had a slight weapons malfunction but, uh everything's perfectly all right now. We're fine. We're all fine here now. Thank you. How are you?"

# Document before tokenization
print(text)

Had a slight weapons malfunction but, uh everything's perfectly all right now. We're fine. We're all fine here now. Thank you. How are you?

word_tokens = word_tokenize(text)

# Document after tokenization - each word is separated out. Compound words like "everything's" are now two words: "everything" and "'s"
print(word_tokens)

['Had', 'a', 'slight', 'weapons', 'malfunction', 'but', ',', 'uh', 'everything', "'s", 'perfectly', 'all', 'right', 'now', '.', 'We', "'re", 'fine', '.', 'We', "'re", 'all', 'fine', 'here', 'now', '.', 'Thank', 'you', '.', 'How', 'are', 'you', '?']

Some terminology

• Tokens: how we’ve broken down the text into smaller units
• Document: the unit of text we’re analysing. Could be sentences, could be paragraphs, could be a whole book. Different breakdowns for different purposes
• Corpus: The collection of documents being analysed

Bag of words

texts = [
    'I love to run',
    'the cat does not eat fruit',
    'run to the cat',
    'I love to eat fruit. fruit fruit fruit fruit'
]

from sklearn.feature_extraction.text import CountVectorizer

count_vectorizer = CountVectorizer()
X = count_vectorizer.fit_transform(texts)
X.toarray()

array([[0, 0, 0, 0, 1, 0, 1, 0, 1],
       [1, 1, 1, 1, 0, 1, 0, 1, 0],
       [1, 0, 0, 0, 0, 0, 1, 1, 1],
       [0, 0, 1, 5, 1, 0, 0, 0, 1]], dtype=int64)

Each column is a different word, and the count vectorizer simply counts how many appearances of each word are in each sentence.

🤔 Can you guess which column represents which word?

It’s column 4: the word “fruit” appears 5 times in the last sentence.

# Visualising what the vectorizer has done

vectorized_texts = pd.DataFrame(
    X.toarray(),
    columns = count_vectorizer.get_feature_names_out(),
    index = texts
)

vectorized_texts

We will try the same code from above - this time on the vectorised text instead of the raw text! This time we shouldn’t get any errors.

naivebayes = MultinomialNB()
count_vectorizer = CountVectorizer()

X = count_vectorizer.fit_transform(data.review)

cv_nb = cross_validate(
    naivebayes,
    X,
    data.sentiment,
    scoring = "accuracy"
)

round(cv_nb['test_score'].mean(),2)

0.85

Our accuracy score is 85% which isn’t too bad

What are the limitations of this approach?

• No context
• Word order not available
• All words treated the same
• Very simplistic approach!

TF-IDF

TERM FREQUENCY (TF)

The more often a word appears in a document relative to others, the more likely it is that it will be important to this document

Example: if a word appears relatively frequently in a document, it is obvious that this word is important to the overall meaning of the document.

texts = [
    'I love to run',
    'the cat does not eat the fruit',
    'run to the cat',
    'I love to eat the fruit. fruit fruit fruit fruit'
]

# In document 4, the Term Frequency (TF) of the word FRUIT is?
# The word fruit appears 5 times
# There are 10 words in the document

5/10

0.5

DOCUMENT FREQUENCY (DF)

If a word appears in many documents of a corpus, it’s not important to understand a particular document.

Example: on eurosport.com/football, the word “football” appears in every article, hence why the word football on this website is an unimportant word!

For the word “football” on Eurosport, we would expect this formula to be close to 1 since the number of docs containing the word “football” will probably only be slightly less than the total number of docs (out of 100 maybe only 5 don’t have the word “football”, so we get 95/100).

if the word “football” appears in all the articles it is not very useful for helping us identify between two articles, but if only a few documents contain words like “concussion” or “wellbeing”, (e.g. they appear in 2/100 articles) it will be much more useful in determining the topic of that article (they are probably specifically about player wellfare).

💡 Thus the intuition of the term frequency - inverse document frequency approach is to give a high weight to any term which appears frequently in a single document, but not in too many documents of the corpus.

## ?? Which words appear frequently in our small corpus 
# and might not be useful for deriving meaning?

texts = [
    'I love to run',
    'the cat does not eat the fruit',
    'run to the cat',
    'I love to eat the fruit. fruit fruit fruit fruit'
]

# the
# to

from sklearn.feature_extraction.text import TfidfVectorizer

# Instantiating the TfidfVectorizer
tf_idf_vectorizer = TfidfVectorizer()

# Training it on the texts
weighted_words = pd.DataFrame(tf_idf_vectorizer.fit_transform(texts).toarray(),
                 columns = tf_idf_vectorizer.get_feature_names_out(),
                index = texts)

weighted_words

Weaknesses of this approach?

• word order still missing
• relationship between words still missing

n-grams

# The two following sentences have the exact same representation in bag of words/ TFIDF approaches
# However, they have very different meanings!

sentences = [
    "I like cats but not dogs",
    "I like dogs but not cats"
]

# Vectorize the sentences
count_vectorizer = CountVectorizer()
sentences_vectorized = count_vectorizer.fit_transform(sentences)

# Show the representations in a nice DataFrame
sentences_vectorized = pd.DataFrame(
    sentences_vectorized.toarray(),
    columns = count_vectorizer.get_feature_names_out(),
    index = sentences
)

# Show the vectorized words
sentences_vectorized

🧑🏻‍🏫 When using a bag-of-words representation, an efficient way to capture context is to consider:

• the count of single tokens (unigrams)
• the count of pairs (bigrams), triplets (trigrams), and more generally sequences of n words, also known as n-grams

4)

😥 With a unigram vectorization, we couldn’t distinguish two sentences with the same words, despite their meaning being quite different

sentences_vectorized

👩🏻‍🔬 What about a bigram vectorization?

# Vectorize the sentences
count_vectorizer_n_gram = CountVectorizer(ngram_range = (1,2)) # BI-GRAMS
sentences_vectorized_n_gram = count_vectorizer_n_gram.fit_transform(sentences)

# Show the representations in a nice DataFrame
sentences_vectorized_n_gram = pd.DataFrame(
    sentences_vectorized_n_gram.toarray(),
    columns = count_vectorizer_n_gram.get_feature_names_out(),
    index = sentences
)

# Show the vectorized movies with bigrams (pairs of words)
sentences_vectorized_n_gram

Word2Vec embeddings

Attempting to capture semantic meaning of words in numerical format

import gensim.downloader

# Lots of different pretrained embeddings we can use for free!
print(list(gensim.downloader.info()['models'].keys()))

['fasttext-wiki-news-subwords-300', 'conceptnet-numberbatch-17-06-300', 'word2vec-ruscorpora-300', 'word2vec-google-news-300', 'glove-wiki-gigaword-50', 'glove-wiki-gigaword-100', 'glove-wiki-gigaword-200', 'glove-wiki-gigaword-300', 'glove-twitter-25', 'glove-twitter-50', 'glove-twitter-100', 'glove-twitter-200', '__testing_word2vec-matrix-synopsis']

# We will use a vectoriser trained on Wikipedia today
model_wiki = gensim.downloader.load('glove-wiki-gigaword-50')

# Vectors based on 2B tweets, 27B tokens, 1.2M vocab!
# 50 dimensions
# N.B. Words not in glove-wiki-gigaword-50 will not have vectors computed. For example, if there was a niche word or acronym like "NHS-R" there would not be a vector for this word.

model_wiki["cat"]

array([ 0.45281 , -0.50108 , -0.53714 , -0.015697,  0.22191 ,  0.54602 ,
       -0.67301 , -0.6891  ,  0.63493 , -0.19726 ,  0.33685 ,  0.7735  ,
        0.90094 ,  0.38488 ,  0.38367 ,  0.2657  , -0.08057 ,  0.61089 ,
       -1.2894  , -0.22313 , -0.61578 ,  0.21697 ,  0.35614 ,  0.44499 ,
        0.60885 , -1.1633  , -1.1579  ,  0.36118 ,  0.10466 , -0.78325 ,
        1.4352  ,  0.18629 , -0.26112 ,  0.83275 , -0.23123 ,  0.32481 ,
        0.14485 , -0.44552 ,  0.33497 , -0.95946 , -0.097479,  0.48138 ,
       -0.43352 ,  0.69455 ,  0.91043 , -0.28173 ,  0.41637 , -1.2609  ,
        0.71278 ,  0.23782 ], dtype=float32)

# King is to Queen as Man is to ...

example_1 = model_wiki["queen"] - model_wiki["king"] + model_wiki["man"]
model_wiki.most_similar(example_1)[0]

('woman', 0.8903914093971252)

# Similar words to cat

model_wiki.most_similar(model_wiki["cat"])

[('cat', 1.0),
 ('dog', 0.9218006134033203),
 ('rabbit', 0.8487820625305176),
 ('monkey', 0.804108202457428),
 ('rat', 0.7891963124275208),
 ('cats', 0.7865270972251892),
 ('snake', 0.7798910140991211),
 ('dogs', 0.7795815467834473),
 ('pet', 0.7792249917984009),
 ('mouse', 0.7731667160987854)]

# Opposite of cold...?

example_2 = model_wiki["good"] - model_wiki["evil"] + model_wiki["cold"]
model_wiki.most_similar(example_2)[0]

('warm', 0.7870427966117859)

Attention mechanism

The basis of transformer-based neural networks like ChatGPT! The paper that started it all: Attention is all you need

1. Each token (word) embedding gets projected ➡️ into 3 further vectors: the query, key and value vectors (usually 768 dimensions each)!!

2. We compute a scaled dot-product 🔴 on the query and key vectors to work out how much each word relates to those around it

3. Take these scores and normalize with softmax ⤵️

4. Multiply by our value vectors ❎, sum and pass to our dense neural network

⚠️ TLDR: The main point is that each word is now represented by 768 * 3 numbers! This is partly what makes LLMs so powerful (and resource-hungry) ⚠️

In the simple bag-of-words and TFIDF approaches, each word was represented by only 1 number each

In more complex word embeddings each word was represented by between 50 to 300 numbers each

Resources

• R text mining book https://www.tidytextmining.com/
• Huggingface tutorials (python) https://huggingface.co/learn/nlp-course/chapter1/1
• Great video on attention https://www.youtube.com/watch?v=zxQyTK8quyY

How to use deploy-pr-preview

We weren’t doing any CI/CD on PRs initially, so first I need to define a new workflow. Workflows are defined in .yaml files in the .github/workflows folder. At the top of the workflow, I need to give it a name and tell it when to trigger. In this case I want it to trigger on any PR.

name: Quarto Preview

on:
  pull_request:
    types:
      - opened
      - reopened
      - synchronize
      - closed

Once I’ve defined when it should run, I need to specify what it should run. That tends to involve a number of steps such as

• Checking out the repository
• Installing system dependencies
• Installing packages (via {renv})
• Rendering the Quarto site

We already have a publish.yml workflow for main which has a number of relevant steps which I’ve borrowed.

The file looks a bit like this:

jobs:
  build-deploy:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ./.github/workflows
    permissions:
      contents: write
      pull-requests: write
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Use cache
        uses: actions/cache@v4
        with:
          key: freeze
          path: _freeze

      - name: Install System Dependencies
        run:  bash install_system_deps.sh

      - name: Set up R
        uses: r-lib/actions/setup-r@v2
        with:
          use-public-rspm: true
      
      - name: Set up renv
        uses: r-lib/actions/setup-renv@v2

      - name: Setup Quarto
        uses: quarto-dev/quarto-actions/setup@v2

      - name: Render
        uses: quarto-dev/quarto-actions/render@v2

Once the site has rendered, we just need to add a step to deploy the PR.

      - name: Deploy PR Preview
        uses: rossjrw/pr-preview-action@v1.4.8
        with:
          source-dir: ./_site/

Now everytime a PR is created, or updated, the GitHub Action bot will spring into action.

Adjusting the publish.yml

That’s all working and looking good! However, there is one last change we need to make. When someone merges into main, the publish action is triggered, and sometimes in that process the pr-previews can get wiped. This means that your preview link would no longer work if someone merges another branch to main whilst you’re working on your PR.

We were using the standard Quarto publish action which as far as I know, does not have an inbuilt configuration to exclude folders from the clean up process.

Instead, I’ve replaced the Quarto action with another marketplace action, github-pages-deploy-action by JamesIves which has an inbuilt way to exclude certain folders from the clean-up. Since it’s not a specific Quarto publish action, we need to remember to add a Quarto render step first.

      - name: Publish to GitHub Pages (and render)
        uses: quarto-dev/quarto-actions/publish@v2
        with:
          target: gh-pages

      - name: Render Quarto
        uses: quarto-dev/quarto-actions/render@v2

      - name: Deploy 🚀
        uses: JamesIves/github-pages-deploy-action@v4
        with:
          folder: _site/
          clean-exclude: pr-preview/
          force: false

Spring cleaning

Now that we have a preview workflow (for branches) and a publish workflow (for main), we have a number of duplicated steps. As any good programmer knows, this is not ideal as it goes against the DRY prinicipal and makes code harder to maintain.

I couldn’t find a quick way for two workflows to share steps¹, so as a small improvement, I moved the lengthy system depencies call into it’s own script. I then ran the script using the following snippet. and called it via the following step.

      - name: Install System Dependencies
        run:  bash install_system_deps.sh

which I think is a bit nicer to read than the original.

 - name: Install System Dependencies
        run: |
          sudo apt update
          sudo apt install -y cmake
          sudo apt install -y gdal-bin
          sudo apt install -y git
          sudo apt install -y libcurl4-openssl-dev
          ...

In order for the actions to find the script, I also specified the path when I defined the job.

jobs:
  build-deploy:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ./.github/workflows

Now we’ve got this working for GitHub pages 🎉 we’d also like to start using it for some of our deployments on Posit Connect. But that’s for another day.

Viewing upcoming sessions

We have created a fully open GitHub project for tracking Coffee & Coding sessions. Any sessions with scheduled dates can be seen in the “Upcoming sessions” view. Clicking on a session title brings up more information, including a brief overview of the session and the people running it. Users with GitHub accounts can make comments or post emoji reactions.

Adding session ideas

To add a session idea:

1. Create a new issue on the data_science repository. Provide a useful title and description for the session.
2. Give your new issue the label C&C☕
3. If you would like to run or contribute to the session, assign yourself to it.
4. Click “Create” to save your session idea as a GitHub issue. You should then be able to see it listed as a “Potential session” on the planner, and others will be able to view, vote for, and comment on your session idea.

Voting for session ideas

We will use thumbs up (👍) emoji reactions to suggested sessions as a voting system to help us with planning and scheduling.

If you see any potential sessions that you are interested in, react to them with a thumbs up emoji. You can see all planned sessions, in order of votes received, listed here.

Example on NHS workforce statistics

The NHS workforce statistics are official statistics published monthly for England.

staff_group <- readRDS(file = "workforce_staff_group.rds")

I want to perform an analysis for each of the 42 integrated care systems (ICS). The {tidyr} nest() function creates a list-column, where each cell contains a mini dataframe for each grouping.

Let’s group by ICS, and call the nested data column raw_data.

group_by_ics <- staff_group |>
    tidyr::nest(raw_data = -ics_name)

The new column is a list-column, with each cell containing an entire tibble of data relating to that individual ICS.

#' echo: false
head(group_by_ics)

# A tibble: 6 × 2
  ics_name             raw_data         
  <chr>                <list>           
1 South East London    <tibble [8 × 6]> 
2 North East London    <tibble [7 × 6]> 
3 North Central London <tibble [12 × 6]>
4 North West London    <tibble [10 × 6]>
5 South West London    <tibble [8 × 6]> 
6 Devon                <tibble [7 × 6]> 

We can grab these mini datasets in the usual way and explore them interactively.

group_by_ics$raw_data[[1]]

# A tibble: 8 × 6
  organisation_name           total hchs_doctors nurses_health_visitors midwives
  <chr>                       <dbl>        <dbl>                  <dbl>    <dbl>
1 Total                       58394         7108                  14939      926
2 Guy's and St Thomas' NHS F… 21361         3003                   6196      281
3 King's College Hospital NH… 13158         2443                   4202      375
4 Lewisham and Greenwich NHS…  6617          979                   2103      271
5 London Ambulance Service N…  7050            4                     44        0
6 NHS South East London ICB     617            9                     43        0
7 Oxleas NHS Foundation Trust  4094          200                   1196        0
8 South London and Maudsley …  5496          471                   1155        0
# ℹ 1 more variable: ambulance_staff <dbl>

Next, let’s apply some simple processing, say converting absolute numbers into percentages, to each of the ICSs in turn.

We use mutate() to create a new list-column staff_percent and map() to apply the processing function to each cell in turn. ¹

#' Convert percent
#' @param raw_staff Tibble containing organisation_name, total and a number of staff categories
#' @return Tibble like raw_staff but with staff categories represented as percentages rather than absolute numbers
convert_percent <- function(staff){
    staff |>
    dplyr::mutate(dplyr::across(.cols = -c(organisation_name, total),
                  .fns =  \(x)x/total)) |>
    dplyr::rename("Doctors" = "hchs_doctors",
                  "Nurses" = "nurses_health_visitors",
                  "Ambulance staff" = "ambulance_staff",
                  "Midwives" = "midwives")
}

processed_staff <-
group_by_ics |>
    dplyr::mutate(
        staff_percent = purrr::map(raw_data, convert_percent)
    )

Where I think this map-and-nest process really comes into its own is creating plots. Often, I find myself wanting to create a couple of different plots for each grouping, and then optionally save the plots with sensible names. Particularly in the analysis stage, I like having these plots in the same row as the raw data, so I can quickly compare and validate.

I’ve created two functions, plot_barchart() and plot_waffle() which take the data and create charts.

#' Plot barchart
#' Makes a bar chart of staff perentages by organisation
#' @param df tibble of staff data in percent format
plot_barchart <- function(df) {
  df |>
    dplyr::filter(organisation_name != "Total") |>
    dplyr::select(-total) |>
    tidyr::pivot_longer(cols = -c(organisation_name), names_to = "job", values_to = "percent") |>
    ggplot2::ggplot(ggplot2::aes(x = percent, y = organisation_name, fill = job)) +
    ggplot2::geom_col(position = "dodge") + 
    ggplot2::scale_x_continuous(labels = scales::percent_format(scale = 100)) +
    ggplot2::labs(x = "", y = "") +
    StrategyUnitTheme::scale_fill_su() + 
    ggplot2::theme_minimal() + 
    ggplot2::theme(legend.title = ggplot2::element_blank())
}

#' Plot waffle
#' Makes a waffle chart to visualise staff breakdown at an ICS level
#' @param raw_staff count data of staff
#' @param title Title for the graphic
plot_waffle <- function(raw_staff, title) {
waffle_data <-
raw_staff |>
    dplyr::filter(organisation_name == "Total") |>
    dplyr::select(-total, -organisation_name) |>
    tidyr::pivot_longer(cols = dplyr::everything(), names_to = "names", values_to = "vals") |>
    dplyr::mutate(vals = round(vals / 100))

ggplot2::ggplot(waffle_data, ggplot2::aes(fill = names, values = vals)) +
  waffle::geom_waffle(n_rows = 8, size = 0.33, colour = "white") +
  ggplot2::coord_equal() +
  ggplot2::theme_void() + 
  ggplot2::theme(legend.title = ggplot2::element_blank()) +
  ggplot2::ggtitle(title)
}

Again, using mutate() I can create a new column called barchart and I can map() the function plot_barchart(), applying it to each row at a time.

graphs <-
processed_staff |>
    dplyr::mutate(
        barchart =  purrr::map(staff_percent, plot_barchart)
    ) 

The resulting column barchart is again a list-column, but this time instead of containing a tibble, it holds a ggplot object. A whole ggplot in a single cell. ²

If we want to pass two arguments to our function, we can replace map() with map2(). Here we’re using map2() to pass the ics_name column to use as a title in our waffle plot. ³

graphs <-
processed_staff |>
    dplyr::mutate(
        waffle =  purrr::map2(raw_data, ics_name, 
            \(data, title) plot_waffle(data, title)
        )
    ) 

Putting it all together

All of these mutate() steps can actually be called in one step. Here’s the full workflow again in full after a little refactor. I’ve also used pivot_longer() to move the two plotting columns into a single plot column. This will make it easier for me to generate nice filenames, and save the plots.

results <-
staff_group |>
    tidyr::nest(raw_data = -ics_name) |>
    dplyr::mutate(
        staff_percent = purrr::map(raw_data, convert_percent),
        barchart =  purrr::map(staff_percent, plot_barchart),
        waffle =  purrr::map2(raw_data, ics_name, \(data, title) plot_waffle(data, title)) 
    )     |>
    tidyr::pivot_longer(cols = c(barchart, waffle), names_to = "plot_type", values_to = "plot") |>
    dplyr::mutate(filename = glue::glue("{snakecase::to_snake_case(ics_name)}_{plot_type}.png"))

The walk() family of functions in {purrr} are used when the function you’re applying does not return an object, but is being used for it’s side-effect, for example reading or writing files.

Here we call walk2(), passing in both the filename column and the plots column are arguments to save all the plots.

purrr::walk2(
  results$filename,
  results$plot,
  \(filename, plot) ggplot2::ggsave(file.path("plots", filename), plot, width = 10, height = 6)
)

By keeping everything together in one nested structure, I personally find it much easier to keep track of my analyses. If you’re doing a more complex or permenant analysis, you might want to consider setting up a more formal data processing pipeline, and following RAP principals.

Coffee & Coding

In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and the slides from the presentation are now available. We discussed the use of Posit Connect Pins and Azure Storage.

In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!

Posit Connect Pins

# A brief intro to using {pins} to store, version, share and protect a dataset
# on Posit Connect. Documentation: https://pins.rstudio.com/


# Setup -------------------------------------------------------------------


install.packages(c("pins", "dplyr")) # if not yet installed

suppressPackageStartupMessages({
  library(pins)
  library(dplyr) # for wrangling and the 'starwars' demo dataset
})

board <- board_connect() # will error if you haven't authenticated before
# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
# • `server` and `key` not provided for `auth = 'manual'`
# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
# • rsconnect package not installed for `auth = 'rsconnect'`
# Run `rlang::last_trace()` to see where the error occurred.

# To authenticate
# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
# Your browser will open to the Posit Connect web page and you're prompted to
# for your password. Enter it and you'll be authenticated.

# Once authenticated
board <- board_connect()
# Connecting to Posit Connect 2024.03.0 at
# <https://connect.strategyunitwm.nhs.uk>

board |> pin_list() # see all the pins on that board


# Create a pin ------------------------------------------------------------


# Write a dataset to the board as a pin
board |> pin_write(
  x = starwars,
  name = "starwars_demo"
)
# Guessing `type = 'rds'`
# Writing to pin 'matt.dray/starwars_demo'

board |> pin_exists("starwars_demo")
# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
# not "starwars_demo".
# [1] TRUE

pin_name <- "matt.dray/starwars_demo"

board |> pin_exists(pin_name) # logical, TRUE/FALSE
board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
board |> pin_browse(pin_name) # view the pin in the browser


# Permissions -------------------------------------------------------------


# You can let people see and edit a pin. Log into Posit Connect and select the
# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
# the 'sharing' options in the 'Access' tab.


# Overwrite and version ---------------------------------------------------


starwars_droids <- starwars |>
  filter(species == "Droid") # beep boop

board |> pin_write(
  starwars_droids,
  pin_name,
  type = "rds"
)
# Writing to pin 'matt.dray/starwars_demo'

board |> pin_versions(pin_name) # see version history
board |> pin_versions_prune(pin_name, n = 1) # remove history
board |> pin_versions(pin_name)

# What if you try to overwrite the data but it hasn't changed?
board |> pin_write(
  starwars_droids,
  pin_name,
  type = "rds"
)
# ! The hash of pin "matt.dray/starwars_demo" has not changed.
# • Your pin will not be stored.


# Use the pin -------------------------------------------------------------


# You can read a pin to your local machine, or access it from a Quarto file
# or Shiny app hosted on Connect, for example. If the output and the pin are
# both on Connect, no authentication is required; the board is defaulted to
# the Posit Connect instance where they're both hosted.

board |>
  pin_read(pin_name) |> # like you would use e.g. read_csv
  with(data = _, plot(mass, height)) # wow!


# Delete pin --------------------------------------------------------------


board |> pin_exists(pin_name) # logical, good function for error handling
board |> pin_delete(pin_name)
board |> pin_exists(pin_name)

Azure Storage in R

You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.

Below is a sample .Renviron file

AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
AZ_STORAGE_CONTAINER=container-name
AZ_TENANT_ID=long-sequence-of-numbers-and-letters
AZ_APP_ID=another-long-sequence-of-numbers-and-letters

install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed

# Load all environment variables
ep_uri <- Sys.getenv("AZ_STORAGE_EP")
app_id <- Sys.getenv("AZ_APP_ID")
container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
tenant <- Sys.getenv("AZ_TENANT_ID")

# Authenticate
token <- AzureAuth::get_azure_token(
  "https://storage.azure.com",
  tenant = tenant,
  app = app_id,
  auth_type = "device_code",
)

# If you have not authenticated before, you will be taken to an external page to
# authenticate!Use your mlcsu.nhs.uk account.

# Connect to container
endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
container <- AzureStor::storage_container(endpoint, container_name)

# List files in container
blob_list <- AzureStor::list_blobs(container)

# If you get a 403 error when trying to interact with the container, you may
# have to clear your Azure token and re-authenticate using a different browser.
# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
# AzureAuth::get_azure_token() step above.

# Upload specific file to container
AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")

# Upload contents of a local directory to container
AzureStor::storage_multiupload(container, "data/*", "newdir")

# Check files have uploaded
blob_list <- AzureStor::list_blobs(container)

# Load file directly from Azure container
df_from_azure <- AzureStor::storage_read_csv(
  container,
  "newdir/cats.csv",
  show_col_types = FALSE
)

# Load file directly from Azure container (by temporarily downloading file
# and storing it in memory)
parquet_in_memory <- AzureStor::storage_download(
  container,
  src = "newdir/cats.parquet", dest = NULL
)
parq_df <- arrow::read_parquet(parquet_in_memory)

# Delete from Azure container (!!!)
for (blobfile in blob_list$name) {
  AzureStor::delete_storage_file(container, blobfile)
}

Azure Storage in Python

This will use the same environment variables as the R version, just stored in a .env file instead.

We didn’t cover this in the presentation, so it’s not in the slides, but the code should be self-explanatory.

import os
import io
import pandas as pd
from dotenv import load_dotenv
from azure.identity import DefaultAzureCredential
from azure.storage.blob import ContainerClient

# Load all environment variables
load_dotenv()
account_url = os.getenv('AZ_STORAGE_EP')
container_name = os.getenv('AZ_STORAGE_CONTAINER')

# Authenticate
default_credential = DefaultAzureCredential()

# Connect to container
container_client = ContainerClient(account_url, container_name, default_credential)

# List files in container - should be empty
blob_list = container_client.list_blob_names()
for blob in blob_list:
    if blob.startswith('newdir'):
        print(blob)

newdir/cats.parquet
newdir/ronald.jpeg

# Upload file to container
with open(file='data/cats.csv', mode="rb") as data:
    blob_client = container_client.upload_blob(name='newdir/cats.csv', 
                                               data=data, 
                                               overwrite=True)

# # Check files have uploaded - List files in container again
blob_list = container_client.list_blobs()
for blob in blob_list:
    if blob['name'].startswith('newdir'):
        print(blob['name'])

newdir/cats.csv
newdir/cats.parquet
newdir/ronald.jpeg

# Download file from Azure container to temporary filepath

# Connect to blob
blob_client = container_client.get_blob_client('newdir/cats.csv')

# Write to local file from blob
temp_filepath = os.path.join('temp_data', 'cats.csv')
with open(file=temp_filepath, mode="wb") as sample_blob:
    download_stream = blob_client.download_blob()
    sample_blob.write(download_stream.readall())
cat_data = pd.read_csv(temp_filepath)
cat_data.head()

# Load directly from Azure - no local copy

download_stream = blob_client.download_blob()
stream_object = io.BytesIO(download_stream.readall())
cat_data = pd.read_csv(stream_object)
cat_data

# !!!!!!!!! Delete from Azure container !!!!!!!!!
blob_client = container_client.get_blob_client('newdir/cats.csv')
blob_client.delete_blob()

blob_list = container_client.list_blobs()
for blob in blob_list:
    if blob['name'].startswith('newdir'):
        print(blob['name'])

newdir/cats.parquet
newdir/ronald.jpeg

ADDENDUM: Accessing data from SharePoint

SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.

Accessing SharePoint requires user authentication, which you’ll be prompted for in the browser when you try to access SharePoint from R. Note that you must ‘follow’ a SharePoint site before you can fetch its content.

To access data on SharePoint, follow these steps:

1. Navigate to the SharePoint page that has the file of interest in it, using your browser.

2. Click the small star in the top right corner of the window, labelled ‘Follow’.

3. Open (or create) your project’s .Renviron file, used for storing environmental variables. You can do this by running usethis::edit_r_environ() from the R console.

4. Save two new environment variables to the .Renviron file:

   1. SP_SITE_NAME: the name of the site (i.e. the name at the top of the browser screen)
   2. SP_FILE_PATH: the full file path from below the Documents/ folder to the file itself, including the file type extension.

5. Save and close the .Renviron file then either restart your R session (CTRL + Shift + F10 on Windows machines) or run readRenviron(".Renviron") to make the new variables available in your session.

6. Use the code below to read in an Excel file (xlsx) into R memory, or adapt it to read other file types.

# Read in the securely saved path and site variables
sharepoint_site <- Sys.getenv("SP_SITE_NAME")
template_path <- Sys.getenv("SP_FILE_PATH")

# access the dataset and save it into a temporary file
site <- Microsoft365R::get_sharepoint_site(sharepoint_site)
drv <- site$get_drive()
tmp_file <- tempfile(fileext = ".xlsx")
drv$download_file(template_path, dest = tmp_file)

data <- readxl::read_xlsx(tmp_file)

#tidy up
unlink(tmp_file)

Official guidance

Posit is the company who build and maintain RStudio. They host a number of cheatsheets on their website, including one for RStudio. They also have a more in-depth user guide.

Command palette

RStudio has a powerful built-in Command Palette, which is a special search box that gives instant access to features and settings without needing to find them in the menus. Many of the tips and tricks we discussed can be found by searching in the Palette. Open it with the keyboard shortcut Ctrl + Shift + P.

For example, let’s say you forgot how to restart R. If you open the Command Palette and start typing ‘restart’, you’ll see the option ‘Restart R Session’. Clicking it will do exactly that. Handily, the Palette also displays the keyboard shortcut (Control + Shift + F10 on Windows) as a reminder.

As for settings, a search for ‘rainbow’ in the Command Palette will find ‘Use rainbow parentheses’, an option to help prevent bracket-mismatch errors by colouring pairs of parentheses. What’s nice is that the checkbox to toggle the feature appears right there in the palette so you can change it immediately.

I refer to menu paths and keyboard shortcuts in the rest of this post, but bear in mind that you can use the Command Palette instead.

Options

In general, most settings can be found under Tools > Global Options… and many of these are discussed in the rest of this post.

But there’s a few settings in particular that we recommend you change to help maximise reproducibility and reduce the chance of confusion. Under General > Basic, uncheck ‘Restore .Rdata into workspace at startup’ and select ‘Never’ from the dropdown options next to ‘Save workspace to .Rdata on exit’. These options mean you start with the ‘blank slate’ of an empty environment when you open a project, allowing you to rebuild objects from scratch¹.

Keyboard shortcuts

You can speed up day-to-day coding with keyboard shortcuts instead of clicking buttons in the interface.

You can see some available shortcuts in RStudio if you navigate to Help > Keyboard Shortcuts Help, or use the shortcut Alt + Shift + K (how meta). You can go to Help > Modify Keyboard Shortcuts… to search all shortcuts and change them to what you prefer².

We discussed a number of handy shortcuts that we use frequently³. You can:

• re-indent lines to the appropriate depth with Control + I
• reformat code with Control + Shift + A
• turn one or more lines into a comment with Control + Shift + C
• insert the pipe operator (%>% or |>⁴) with Control + Shift + M⁵
• insert the assignment arrow (<-) with Alt + - (hyphen)
• highlight a function in the script or console and press F1 to open the function documentation in the ‘Help’ pane
• use ‘Find in Files’ to search for a particular variable, function or string across all the files in your project, with Control + Shift + F

Themes

You can change a number of settings to alter RStudio’s theme, colours and fonts to whatever you desire.

You can change the default theme in Tools > Global Options… > Appearance > Editor theme and select one from the pre-installed list. You can upload new themes by clicking the ‘Add’ button and selecting a theme from your computer. They typically have the file extension .rsthemes and can be downloaded from the web, or you can create or tweak one yourself. The {rsthemes} package has a number of options and also allows you to switch between themes and automatically switch between light and dark themes depending on the time of day.

In the same ‘Appearance’ submenu as the theme settings, you can find an option to change fonts. Monospace fonts, ones where each character takes up the same width, will appear here automatically if you’ve installed them on your computer. One popular font for coding is Fira Code, which has the special property of converting certain sets of characters into ‘ligatures’, which some people find easier to read. For example, the base pipe will appear as a rightward-pointing arrow rather than its constituent vertical-pipe and greater-than symbol (|>).

Panes

# Header ------------------------------------------------------------------

## Section ----

### Subsection ----

Magic wand

There’s a miscellany of useful tools available when you click the ‘magic wand’ button in the script pane.

This includes:

• ‘Rename in Scope’, which is like find-and-replace but you only change instances with the same ‘scope’, so you could select the variable x, go to Rename in Scope and then you can edit all instances of the variable in the document and change them at the same time (e.g. to rename them)
• ‘Reflow Comment’, which you can click after higlighting a comments block to have the comments automatically line-break at the maximum width
• ‘Insert Roxygen Skeleton’, which you can click when your cursor is inside the body of a function you’ve written and a {roxygen2} documentation template will be added above your function with the @params argument names pre-filled

Along with ‘Comment/Uncomment Lines’, ‘Reindent Lines’ and ‘Reformat Lines’, mentioned above in the keyboard shortcuts section.

Wrapping up

Time was limited in our discussion. There are so many more tips and tricks that we didn’t get to. Let us know what we missed, or what your favourite shortcuts and settings are.

Sankey plot

We now have a neat little summary of movements of participants between the milestones in our recruitment pathway. However, this ‘tidy’ data isn’t the format required by plotly, so the next steps are to prepare it ready for plotting.

df_sankey <- df_flows |>
  # convert our flows data to a tidy graph object
  as_tbl_graph()

df_sankey |>
  activate(what = "nodes") |>
  as_tibble() |>
  reactable(defaultPageSize = 5)

df_sankey |>
  activate(what = "edges") |>
  as_tibble() |>
  reactable(defaultPageSize = 5)

# extract the nodes to a dataframe
nodes <- df_sankey |>
  activate(nodes) |>
  data.frame() |>
  mutate(
    id = row_number() - 1
  )

# extract the edges to a dataframe
edges <- df_sankey |>
  activate(edges) |>
  data.frame() |>
  mutate(
    from = from - 1,
    to = to - 1
  )

# plot our sankey
plot_ly(
  # setup
  type = "sankey",
  orientation = "h",
  arrangement = "snap",

  # use our node data
  node = list(
    label = nodes$name
  ),

  # use our link data
  link = list(
    source = edges$from,
    target = edges$to,
    value = edges$flow
  )
)

# get the eligible population as a single value
# NB, will be used to work out % amounts in each node and edge
temp_eligible_pop <- df_flows |>
  filter(from == "Eligible population") |>
  summarise(total = sum(flow, na.rm = T)) |>
  pull(total)

# style our nodes object
nodes <- nodes |>
  mutate(
    # colour ----
    # add colour definitions, green for accepted, red for declined
    colour = case_when(
      str_detect(name, "Accepted") ~ "#44bd32",
      str_detect(name, "Declined") ~ "#c23616",
      str_detect(name, "No response") ~ "#7f8fa6",
      str_detect(name, "Eligible population") ~ "#7f8fa6"
    ),

    # add a semi-transparent colour for the edges based on node colours
    colour_fade = col2hcl(colour = colour, alpha = 0.3),

    # positioning ----
    # NB, I found that to position nodes you need to supply both
    # horizontal and vertical positions
    # NNB, it was a bit of trial and error to get the these positions just
    # right

    # horizontal positions (0 = left, 1 = right)
    x = case_when(
      str_detect(name, "Eligible population") ~ 1,
      str_detect(name, "Invitation 1") ~ 2,
      str_detect(name, "Invitation 2") ~ 3,
      str_detect(name, "Invitation 3") ~ 4,
      .default = 5
    ) |> rescale(to = c(0.001, 0.9)),

    # vertical position (1 = bottom, 0 = top)
    y = case_when(
      str_detect(name, "Eligible population") ~ 5,
      # invite 1
      str_detect(name, "Invitation 1 Accepted") ~ 1,
      str_detect(name, "Invitation 1 No response") ~ 5,
      str_detect(name, "Invitation 1 Declined") ~ 8.5,
      # invite 2
      str_detect(name, "Invitation 2 Accepted") ~ 2,
      str_detect(name, "Invitation 2 No response") ~ 5,
      str_detect(name, "Invitation 2 Declined") ~ 7.8,
      # invite 3
      str_detect(name, "Invitation 3 Accepted") ~ 2.7,
      str_detect(name, "Invitation 3 No response") ~ 5.8,
      str_detect(name, "Invitation 3 Declined") ~ 7.2,
      # final outcomes
      str_detect(name, "Accepted") ~ 1,
      str_detect(name, "No response") ~ 5,
      str_detect(name, "Declined") ~ 8,
      .default = 5
    ) |> rescale(to = c(0.001, 0.999))
  ) |>
  # add in a custom field to show the percentage flow
  left_join(
    y = df_flows |>
      group_by(to) |>
      summarise(
        flow = sum(flow, na.rm = T),
        flow_perc = percent(flow / temp_eligible_pop, accuracy = 0.1),
      ) |>
      select(name = to, flow_perc),
    by = "name"
  )

# view our nodes data
nodes |>
  reactable(defaultPageSize = 5)

edges <- edges |>
  mutate(
    # add a label for each flow to tell us how many people are in each
    label = number(flow, big.mark = ","),
    # add a percentage flow figure
    flow_perc = percent(flow / temp_eligible_pop, accuracy = 0.1)
  ) |>
  # add the faded colour from our nodes object to match the destinations
  left_join(
    y = nodes |> select(to = id, colour_fade),
    by = "to"
  )

# view our edges data
edges |>
  reactable(defaultPageSize = 5)

# plot our stylised sankey
plot_ly(
  # setup
  type = "sankey",
  orientation = "h",
  arrangement = "snap",

  # use our node data
  node = list(
    label = nodes$name,
    color = nodes$colour,
    x = nodes$x,
    y = nodes$y,
    customdata = nodes$flow_perc,
    hovertemplate = "%{label}<br /><b>%{value}</b> participants<br /><b>%{customdata}</b> of eligible population"
  ),

  # use our edge data
  link = list(
    source = edges$from,
    target = edges$to,
    value = edges$flow,
    label = edges$label,
    color = edges$colour_fade,
    customdata = edges$flow_perc,
    hovertemplate = "%{source.label} → %{target.label}<br /><b>%{value}</b> participants<br /><b>%{customdata}</b> of eligible population"
  )
) |>
  layout(
    font = list(
      family = "Arial, Helvetica, sans-serif",
      size = 12
    ),
    # make the background transparent (also removes the text shadow)
    paper_bgcolor = "rgba(0,0,0,0)"
  ) |>
  config(responsive = T)

Conclusion

Creating Sankey plots in R using plotly is an effective way to visualise patient pathways.

In our project we embedded Sankey plots within an interactive Shiny app which allows for selective filters that update the resulting plot. This allowed us to quickly compare the effects of different models of delivering the screening programme, geography, deprivation levels, patient demographic, or any combination of these.

Their use has helped the programme team better understand patient flows through the pathway, where the points of drop-off are and compare / contrast the effects of different models of delivering the screening programme on patient engagement.

Feedback from external stakeholders has been positive too, noting how easy it is to engage with and understand this style of presentation.

In this blog post we have wrangled a dataset to describe how people flow between steps in a process and then produced a Sankey diagram with some stylistic touches to make an effective visualisation.

I hope this post helps you feel better prepared to use Sankeys in your work.