Computer Vision

Is this AI?

10 October 2024

What is Computer Vision (CV)

Computer vision is a field of computer science that focuses on enabling computers to identify and understand objects and people in images and videos.

Like other types of AI, computer vision seeks to perform and automate tasks that replicate human capabilities.

In this case, computer vision seeks to replicate both the way humans see, and the way humans make sense of what they see.

What tasks can CV be used for?

Classification

Assign a single label to each image

Dog

Welsh Spaniel

Animal in water

Dog

Sussex Spaniel

Animal on land

Multi-Classification

Assign one or more labels to each image

Dog, Welsh Spaniel, Animal in water

Dog, Sussex Spaniel, Animal on land

Object Detection

Event Detection

How does CV work?

for classification tasks

get a very large corpus or labelled images
convert the images to a form that the computer can work with (tensors)
feed into a convolutional neural network
profit?

Large corpus of labelled images

ImageNet

a large visual database
over 14 million hand-annotated images
more than 20,000 categories
each category has “several hundred” images
started in 2006

Convert the images to tensors

A fancy way of saying:

turn the images into a 2d table

of values between 0 and 1

(Convolutional) Neural Networks

Use pre-trained models

Models that have been pre-trained on some image datasets which can be downloaded and used

Transfer learning is the concept of taking a pre-trained model as a basis, then fine-tuning it to classify based on your own images.

How can CV be used in Healthcare?

classification
multi-classification
object detection
event detection

How can CV be used in Healthcare?

detecting disease or injury
monitoring patients vitals, e.g. respiratory rate
detecting bounds of a tumour when planning radiotherapy
automating cell counting
calculating the grade of a cancer
monitor for long-term changes, e.g. AAA
devices for patients with vision impairments
detecting when patients move (fall prevention)
monitoring the hygiene of a ward

Issues with Computer Vision

… one neural network learned to differentiate between dogs and wolves. It didn’t learn the differences between dogs and wolves, but instead learned that wolves were on snow in their picture and dogs were on grass.

Issues with Computer Vision

algorithm trained at Mount Sinai Hospital, New York City
Busy ICU, many elderly patients
34% of their x-rays came from patients with pneumonia
93% accuracy

tested at other sites, pneumonia ~1% of x-rays
accuracy dropped to 73%-80%

Issues with Computer Vision

At Mount Sinai, many of the infected patients were too sick to get out of bed, and so doctors used a portable chest x-ray machine. Portable x-ray images look very different from those created when a patient is standing up. Because of what it learned from Mount Sinai’s x-rays, the algorithm began to associate a portable x-ray with illness. It also anticipated a high rate of pneumonia.

The Unique Problems of Medical Computer Vision

This is the very unique problem of medical computer vision: we are attempting to solve a small signal on the background of small noise whereas standard computer vision’s problem is a large signal on the background of large noise.

The Unique Problems of Medical Computer Vision

Is this a dog?

English Springer Spaniel in a Field, Wikipedia

Small-cell carcinoma of the lung, Wikipedia

Annotations of cats & dogs is cheaper than reviewing medical scans/slides. The latter adds an additional burden on health systems.

Other issues with CV

Early detection vs over diagnosis
Adversarial attacks can trick CV algorithms to incorrectly classify images
Computational power (environmental impact)
Governance: have we got consent to use images?
Explainability of Neural Networks