Framework development

How the framework was built

The inclusion of each sub-category in the framework was based on a robust evidence-based rationale, drawing on multiple strands of work, including research with patient experience and quality improvement teams across different types of 19 NHS Trusts, exploration of pre-existing frameworks including the one used in the phase 1 of this project, manual qualitative inductive coding, and manual data labelling.

The initial version of the framework was built using a ground-up qualitative inductive coding approach. This consisted of a qualitative analyst manually reading a large volume of patient experience qualitative data and devising categories as they emerged in the data. This approach has meant that the sub-categories were:

• All seen in patient experience qualitative data.
• Reflective of the issues patients themselves raise, how they present their experiences and the language they use (rather than attempting to fit experiences into pre-defined categories which would be a deductive rather than an inductive approach).
• Considered to be distinct enough to form a unique sub-category in terms of vocabulary used and emphasis, thereby aiding their viability for an automated model.
• Collectively able to result in an overall framework which makes comprehensive use of the data.

As the project progressed a large volume of qualitative data from across different trust types was manually labelled by qualitative analysts (i.e., assigned to the relevant sub-categories), to produce the dataset that was used to train and test the machine learning model. During this time further iterations were made to the framework as follows:

• A new sub-category was added as qualitative analysts identified that an additional topic was emerging in the data which had not already been seen.
• Small amendments were made to category/ sub-category names and descriptions to ensure they remained well aligned to the qualitative data and reflected any nuances seen across different trust types.
• Some sub-categories were merged because they were low in volume (negatively impacting the models’ ability to identify them), there was a high degree of overlap with other sub-topics and/or thematic similarity in content.
• Some sub-categories were removed because they were extremely low volume (negatively impacting the models’ ability to identify them), the content was already sufficiently captured by other sub-categories, there was a lack of thematic coherence across comments, or the model did not perform well enough for them to be included.

Due to the way that it was built the final framework is one that is qualitatively grounded in the data, but also works well from a data science perspective.