Getting started

azkit is an R package to help you:

• handle authentication with Azure
• perform basic tasks accessing Azure blob storage and table storage

The purpose of this page is to show you a few ways you might use {azkit} functions to achieve certain tasks. It’s not the only way you can set things up and use the functions, it’s just an opinionated example workflow - you may choose to do things differently 😀.

Authentication with Azure storage

This is an important first step.

azkit supports various approaches to authentication and authorisation, depending on the kind of resource you need to access.

The get_auth_token() function therefore supports various options, for example whether to generate a v1 or v2 token, whether the code is running on a managed resource or locally, whether to use a locally-stored secret or an authorisation code, and so on. Our function is a wrapper around functions in the AzureAuth package, and reading the documentation provided for that package is a good way to explore what specific approaches might be required for specific circumstances.

For standard users, however, the default options provided by our function should be sufficient and should be the simplest way to get up and running.

As a local user (not running on a managed resource), you first of all need to get an authentication token that verifies to Azure that you have a certain user account.

Do this from your computer when online, as it requires user interaction - it will require you to log in via your web browser, or via a Microsoft login app. R cannot do this bit for you 😊.

Run

azkit::get_auth_token()

at the R console. Alternatively, install the Azure CLI tool and do:

az login

at the terminal. This should pop up a window asking you to authenticate with Azure online - beware, sometimes this window is not immediately visible, it may be hidden behind other windows on your screen.

The authentication token will be stored under your user account files on your computer.

You can see which tokens you have by running

AzureAuth::list_azure_tokens()

See also the Troubleshooting vignette.

Refreshing your auth token

In a data analysis workflow, if you have done something like:

token <- azkit::get_auth_token()

and the token later expires*, you should be able to just do this again:

token <- azkit::get_auth_token()

but alternatively you can use

azkit::refresh_token(token)

If these don’t work, you can force azkit to go get you a whole new fresh token from the internet by doing:

token <- get_auth_token(force_refresh = TRUE)

* auth tokens have a limited validity period but are supposed to auto-refresh when needed

Setting up environment variables

Use an .Renviron file to set 2 required environment variables (envvars). These are the endpoint URLs of your Azure storage account, for blob storage and table storage.

There are two ways you can set variables using .Renviron files:

• Globally for your local (eg Windows) user account
• Project-specific

The usethis packages provides a handy helper to create/edit these files:

To use the first, global option (probably the most straightforward):

usethis::edit_r_environ()

or to edit a project-specific .Renviron file, while within the project folder:

usethis::edit_r_environ(scope = "project")

The latter makes sense if you have multiple projects that may be using different Azure endpoints, or a specific project that uses a different account to your usual one. Otherwise it is probably simpler to set the variables globally and then you can basically forget about it!

⚠️ Important

If using a project-specific file, ensure this is listed in your .gitignore file

What to add to your .Renviron file

For the purposes of this package, the file just needs to contain two endpoint URLs with the following names:

AZ_STORAGE_EP=[your azure blob storage endpoint url]
AZ_TABLE_EP=[your azure table storage endpoint url]

See the .Renviron.example file in the azkit GitHub repository.

For use within The Strategy Unit, ask a member of the Data Science team to send you the necessary values for these URLs.

Changes to the .Renviron file will require you to reload your environment in order to update it, such as by restarting your R session.

azkit is expecting these variables to be present in your working environment. The reason we set these as environment variables is that the URLs are generally not meant to be made public, for security reasons. Referencing them as envvars means that their actual values do not need to be included explicitly in any code that might be made public.

💡 Tip

If you use a project .Renviron file as well as a global one, the variables in the project file will override those in the global file. Ensure that any project .Renviron is not completely blank, as this will unset any variables from the global file.

Working with Azure blob storage

Containers

Reading in data from Azure blob storage requires you to first set up access to a particular container where your data is located.

The azkit::get_container() function makes this easy.

You may have a container name stored as an environment variable, for security. If not, just provide the name of the container as a string instead.

First read this in and then use it to create a container object in R:

container_name <- Sys.getenv("AZ_STORAGE_DATA_CONTAINER")
# or as a string:
# container_name <- "example-data"
data_container <- azkit::get_container(container_name)

If you don’t know the names of the containers you have access to, you can use:

azkit::list_container_names()

to output a list.

💡 Tip

Alternatively, it can be useful to browse your Azure storage account on the web at https://portal.azure.com/ or use Azure Storage Explorer.

Accessing files in the container

Use azkit::list_files() to see what files are available.

Re-using our data_container variable from the code above, we might do something like:

azkit::list_files(data_container, "my-data-folder")

This will return a list of the all the files in that specific directory. By default this is non-recursive: it does not return any files within sub-directories. You can change this behaviour by using the recursive argument, but beware that you might end up with a lot of filenames if there is lots of data stored in a sub-directory structure!

azkit::list_files(data_container, "my-data-folder", recursive = TRUE)

You can also limit list_files to only return files with a certain filetype extension (no . required before the extension). (By default it will list all files.)

azkit::list_files(data_container, "my-data-folder", ext = "json")

Once you have your filename, you can use one of a selection of azkit functions to help read the data into R.

azkit comes with five functions optimised for reading in data from:

• parquet (azkit::read_azure_parquet())
• json (azkit::read_azure_json())
• json.gz (azkit::read_azure_jsongz())
• csv (azkit::read_azure_csv())
• rds (azkit::read_azure_rds())

These come with helper functions already set up to read the data into R in the most convenient way. There is also scope to customise how these functions operate - see each function’s help page for more.

For example, azkit::read_azure_csv() allows you to pass additional options through to readr::read_delim, such as the col_types argument.

There is also the azkit::read_azure_file() function that will attempt to read from any given file, for situations not covered by the above 5. But you will need to handle the output returned by this yourself.

Example: reading a parquet file

parquet_data <- azkit::read_azure_parquet(data_container, "data/info.parquet")

Example: reading a CSV file

Simplest method:

csv_data <- azkit::read_azure_csv(data_container, "csv_data/info.csv")

With additional options passed through:

csv_data <- data_container |>
  azkit::read_azure_csv("csv_data/info.csv", col_types = "cc-ii")

csv_data <- azkit::read_azure_csv(
  data_container,
  "csv_data/info.csv",
  col_select = tidyselect::starts_with("active")
)

Example: reading multiple files

The azkit::read_azure_* functions don’t read in multiple files by default, but this can be achieved by using them within an apply or map operation:

csv_files <- azkit::list_files(data_container, "csv_data", ext = "csv")
csv_names <- tools::file_path_sans_ext(basename(csv_files))
csv_data_list <- csv_files |>
  purrr::map(\(x) azkit::read_azure_csv(data_container, x)) |>
  rlang::set_names(csv_names) # optional, may be useful

Working with Azure table storage

Table storage does not require access to containers. You just need the name of the table you want to access.

We will use the azkit::read_azure_table() function.

You may have a table name stored as an environment variable, for security. If not, just provide the name of the table as a string instead.

To read in the entire table:

table_name <- Sys.getenv("AZ_TABLE_NAME")
# or as a string:
# table_name <- "example-table"
table_data <- azkit::read_azure_table(table_name)

Within this function, you can filter the table that is returned by using OData clauses (“system query options”). This is especially useful with large tables where you wish to improve speed by only requesting certain rows or columns. Use the filter, select, and/or top arguments to achieve this.

Example of the OData syntax

table_data <- azkit::read_azure_table(
  table_name,
  filter = "Status eq 'Active' and Score ge 10", # filter rows by variable value
  select = "PartitionKey,RowKey,Status,Score", # only return certain columns
  top = 100 # only return the first 100 rows
)

A very initial guide to how to construct your own clauses (query options) can be found on Microsoft Learn. Although the examples there are not written in R, the syntax for the query options should be in a valid format for what you need to supply to azkit::read_azure_table().