An R package to handle Azure authentication and some basic tasks accessing blob and table storage and reading in data from files.
Status
The package is stable and useable. Please create a GitHub issue if you encounter problems, or have ideas for its improvement.
Installation
You can install azkit with:
# install.packages("pak")
pak::pak("The-Strategy-Unit/azkit")Usage
A primary function in azkit enables access to an Azure blob container:
# This assumes you have Azure authentication working, and the correct
# Azure endpoint variables in your environment (see below).
# NB the file/data locations used here are invented examples.
data_container <- azkit::get_container("data-container")Authentication should be handled automatically by get_container(), but if you need to, you can explicitly return an authentication token for inspection or re-use:
my_token <- azkit::get_auth_token()(For issues with authentication, including mysterious error messages, please see the Troubleshooting section below or the Troubleshooting vignette.)
data_container <- azkit::get_container("data-container", token = my_token)Return a list of all available containers in your default Azure storage with:
azkit::list_container_names()Once you have access to a container, you can use one of a set of data reading functions to bring data into R from .parquet, .rds, .json or .csv files. For example:
pqt_data <- azkit::read_azure_parquet(data_container, "important_data.parquet")To read in any file from the container in raw format, to be passed to the handler of your choice, use:
raw_data <- azkit::read_azure_file(data_container, "misc_data.ext")Currently these functions only read in a single file at a time.
You can map over multiple files by first using azkit::list_files() and then passing the file paths to the read* function:
azkit::list_files(data_container, "data/latest", "parquet") |>
purrr::map(\(x) azkit::read_azure_parquet(data_container, x))You can also pass through arguments in ... that will be applied to the appropriate handler function (see documentation). For example, readr::read_delim() is used under the hood by azkit::read_azure_csv, so you can pass through an argument such as col_types:
csv_data <- data_container |>
azkit::read_azure_csv("vital_data.csv", path = "data", col_types = "ccci")Environment variables
To facilitate access to Azure Storage you should set some environment variables. The neatest way to do this is to include a .Renviron file in your project folder.
⚠️ These values are sensitive and should not be exposed to anyone outside The Strategy Unit. Make sure you include .Renviron in the .gitignore file for your project.
Your .Renviron file can contain the variables below. Ask a member of the Data Science team for the necessary values.
AZ_STORAGE_EP=
AZ_TABLE_EP=Troubleshooting
Azure authentication is probably the main area where you might experience difficulty.
[!NOTE] Users at The Strategy Unit: when initially authenticating with Azure, you will be sent to the web. Since you may be logged into one of multiple Microsoft tenants, it is important that you authenticate with the same account (MLCSU) that you use for access to Azure storage. If you authenticate when logged into your nhs.net Microsoft account in your browser, your token will not be useable for access to Azure.
To debug, try running:
azkit::get_auth_token()and see what is returned.
AzureRMR::get_azure_login() or AzureRMR::list_azure_tokens() may also be helpful for troubleshooting - try them and see if they work / what they reveal.
To refresh a token, you can do:
# if previously you did:
# token <- azkit::get_auth_token()
azkit::refresh_token(token)If you get errors when reading in files, first check that you are passing in the full and correct filepath relative to the root directory of the container.
If read_azure_json() and similar are not working as expected, try reading in the raw data first with azkit::read_azure_file() and then passing that to a handler function of your choice.
Getting help
Please use the Issues feature on GitHub to report any bugs, ideas or problems, including with the package documentation.
Alternatively, to ask any questions about the package you may contact Fran Barton.