# 🧩 3.5 Computer Vision

```{contents}
:depth: 3
```

## 🔰 Tutorial

In this module, you will develop software to:
1. Use **OpenCV** for image processing and object detection
2. Implement spatial referencing and ID lookup using **AprilTags**
3. Automate a motorized microscope for sample analysis using **OpenCV**

### 1. OpenCV Basics

OpenCV (Open Source Computer Vision Library) is a powerful and versatile open-source computer vision and machine learning software library. It is widely used for a variety of image and video processing tasks, making it an essential tool for developers working on computer vision applications.

Key features and capabilities of OpenCV include:

1. Image Processing: Basic operations like filtering, color space conversion, and geometric transformations.
2. Object Detection: Identifying and locating objects within images or video streams.
3. Feature Detection and Description: Identifying distinctive features in images for tasks like image matching and stitching.
4. Video Analysis: Processing video streams, including motion detection and object tracking.
5. Camera Calibration: Correcting lens distortion and determining camera position.
6. 3D Reconstruction: Creating 3D models from 2D images.
7. Machine Learning: Implementing various machine learning algorithms for classification and clustering tasks.
8. GPU Acceleration: Utilizing GPU processing power for faster computations.

OpenCV is cross-platform and supports multiple programming languages, including Python, C++, and Java. Its extensive functionality and active community make it a go-to choice for both beginners and experienced developers in the field of computer vision.

#### Bill of Materials

- [OpenCV](https://pypi.org/project/opencv-python/) (for Python installation: `pip install opencv-python`)
- [AprilTag Python Library](https://pypi.org/project/apriltag/)
- [Motorized Microscope]
- [Raspberry Pi Camera Module](https://www.raspberrypi.org/products/camera-module-v2/)
- [USB-A to micro USB-B cable]
- Printed AprilTags (can be generated and printed from [AprilTag Generation](https://github.com/AprilRobotics/apriltag-generation))


#### Demo

✅ Read the [OpenCV Documentation](https://docs.opencv.org/4.x/)

✅ Watch [Introduction to OpenCV](https://www.youtube.com/watch?v=oXlwWbU8l2o)

In this task, you will use **OpenCV** to perform image preprocessing and identify regions of interest (ROI).

```python
import cv2
import numpy as np


def preprocess_image(image):
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    blurred = cv2.GaussianBlur(gray, (5, 5), 0)
    thresh = cv2.adaptiveThreshold(
        blurred, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 11, 2
    )
    return thresh


def find_roi(image):
    processed = preprocess_image(image)
    contours, _ = cv2.findContours(
        processed, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
    )
    contours = sorted(contours, key=cv2.contourArea, reverse=True)

    rois = []
    for contour in contours[:5]:
        area = cv2.contourArea(contour)
        perimeter = cv2.arcLength(contour, True)
        circularity = 4 * np.pi * area / (perimeter * perimeter)
        if circularity > 0.7 and area > 1000:
            x, y, w, h = cv2.boundingRect(contour)
            rois.append((x, y, w, h))
    return rois


def main():
    image = cv2.imread("sample_image.jpg")
    if image is None:
        print("Error: Could not load image")
        return

    rois = find_roi(image)
    for roi in rois:
        x, y, w, h = roi
        cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)

    cv2.imshow("ROI Detection", image)
    cv2.waitKey(0)
    cv2.destroyAllWindows()


if __name__ == "__main__":
    main()
```

### 2. AprilTags for Spatial Referencing

AprilTags are a popular fiducial marker system used for reliable and robust spatial referencing and object identification in robotics and computer vision applications. These tags are particularly useful in scenarios where precise localization and orientation information is required.

Key features and applications of AprilTags in spatial referencing include:

1. Precise Localization: AprilTags provide accurate position and orientation information in 3D space.
2. Robust Detection: They are designed to be easily detectable even under varying lighting conditions and perspectives.
3. Unique Identification: Each tag has a unique ID, allowing for multiple tag tracking in a single scene.
4. Scale Invariance: AprilTags can be detected at various distances, making them suitable for different spatial scales.
5. Low Computational Cost: The detection algorithm is efficient, allowing for real-time processing.
6. Augmented Reality: They are often used as anchors for AR applications.
7. Robot Navigation: AprilTags can serve as landmarks for robot localization and mapping.
8. Camera Calibration: They can be used to assist in camera calibration processes.

In spatial referencing, AprilTags act as known reference points in the environment. By detecting these tags and calculating their pose (position and orientation) relative to the camera, we can:

- Determine the camera's position in the environment
- Measure distances and angles between objects
- Create local coordinate systems for precise manipulation tasks
- Enable visual servoing for robotic systems

This makes AprilTags an invaluable tool in robotics, computer vision, and augmented reality applications where accurate spatial information is crucial.

#### Bill of Materials
- [AprilTag Python Library](https://pypi.org/project/apriltag/)
- [AprilTags Documentation](https://april.eecs.umich.edu/software/apriltag.html)
- Printed AprilTags (can be generated and printed from [AprilTag Generation](https://github.com/AprilRobotics/apriltag-generation))

#### Demo

✅ Watch [Vision Programming with AprilTags](https://youtu.be/TG9KAa2EGzQ)

The following code demonstrates how to use **AprilTags** for spatial referencing:

```python
import cv2
import numpy as np
from pupil_apriltags import Detector


def detect_apriltags(image):
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    at_detector = Detector(
        families="tag36h11",
        nthreads=1,
        quad_decimate=1.0,
        quad_sigma=0.0,
        refine_edges=1,
        decode_sharpening=0.25,
        debug=0,
    )
    tags = at_detector.detect(gray)
    return tags


def spatial_referencing(image, tags, tag_size=0.05):
    fx, fy, cx, cy = 1000, 1000, image.shape[1] / 2, image.shape[0] / 2
    camera_params = [fx, fy, cx, cy]

    for tag in tags:
        pose, e0, e1 = tag.fit_pose(camera_params, tag_size)
        rotation_matrix = pose[:3, :3]
        translation_vector = pose[:3, 3]
        euler_angles = cv2.Rodrigues(rotation_matrix)[0].flatten()

        cv2.polylines(image, [np.int32(tag.corners)], True, (0, 255, 0), 2)
        center = tuple(map(int, tag.center))
        cv2.circle(image, center, 5, (0, 0, 255), -1)
        cv2.putText(
            image,
            str(tag.tag_id),
            (center[0] - 10, center[1] - 10),
            cv2.FONT_HERSHEY_SIMPLEX,
            0.5,
            (0, 0, 255),
            2,
        )

        print(f"Tag ID: {tag.tag_id}")
        print(
            f"Position: X={translation_vector[0]:.2f}, Y={translation_vector[1]:.2f}, Z={translation_vector[2]:.2f}"
        )
        print(
            f"Rotation: Roll={np.degrees(euler_angles[0]):.2f}, Pitch={np.degrees(euler_angles[1]):.2f}, Yaw={np.degrees(euler_angles[2]):.2f}"
        )
        print("---")

    return image


# Main function for AprilTag detection and spatial referencing
def main_apriltag():
    cap = cv2.VideoCapture(0)
    while True:
        ret, frame = cap.read()
        if not ret:
            print("Failed to capture image")
            break

        tags = detect_apriltags(frame)
        frame_with_tags = spatial_referencing(frame, tags)
        cv2.imshow("AprilTag Spatial Referencing", frame_with_tags)

        if cv2.waitKey(1) & 0xFF == ord("q"):
            break

    cap.release()
    cv2.destroyAllWindows()


if __name__ == "__main__":
    main_apriltag()
```

This example code performs the following tasks:

1. **AprilTag Detection (`detect_apriltags` function)**:
   - Converts the input image to grayscale.
   - Uses the `pupil_apriltags` Detector to identify AprilTags in the image.
   - Returns a list of detected tags.

2. **Spatial Referencing (`spatial_referencing` function)**:
   - For each detected tag:
     - Calculates the 3D pose (position and orientation) of the tag relative to the camera.
     - Extracts the translation vector (position) and rotation matrix.
     - Converts the rotation matrix to Euler angles for easier interpretation.
     - Draws the tag outline, center point, and ID on the image.
     - Prints the tag's ID, position, and orientation.

3. **Main Loop (`main_apriltag` function)**:
   - Captures video frames from the camera.
   - Detects AprilTags in each frame.
   - Performs spatial referencing on the detected tags.
   - Displays the processed frame with tag information.
   - Continues until the user presses 'q' to quit.

This code demonstrates how AprilTags can be used for real-time spatial referencing in a video stream, providing precise position and orientation information for each detected tag relative to the camera.

### 3. Motorized Microscope Automation with OpenCV

In this section, we'll combine OpenCV image processing techniques with a motorized microscope to implement automated monitoring of microscopic particles.

#### Bill of Materials

- OpenFlexure Microscope (https://openflexure.org/)
  The OpenFlexure Microscope is a customisable, open-source optical microscope, using either very cheap webcam optics or lab quality, RMS threaded microscope objectives. It uses an inverted geometry, and has a high quality mechanical stage which can be motorised using low cost geared stepper motors.

#### Demo

✅ Watch
[Building the OpenFlexure Microscope](https://www.youtube.com/watch?v=aQEyoch3iuo&ab_channel=TinkerTechTrove)

This example code combines OpenCV image processing techniques with simulated microscope functionality to implement automated monitoring of microscopic particles. The main features include:
1. Image preprocessing: Using Gaussian blur and adaptive thresholding to enhance image quality.
2. Particle detection: Using contour detection methods to identify potential microscopic particles.
3. Microscope scanning: Utilizing the simulated microscope to scan an area and obtain multiple images.
4. Growth or distribution analysis: Calculating the number of detected particles and estimating changes over time.
5. Visualization: Marking detected particles on each scanned image and displaying the results.

This program performs periodic scanning and analysis to continuously monitor the sample. You can adjust the scanning interval, particle detection parameters, and other settings according to your specific requirements and sample type.

Note: The visibility and detection of particles will depend on the microscope's magnification and the size of the particles you're observing. This example can be used for various types of samples, such as:
- Microspheres (1-100 μm)
- Pollen grains (10-100 μm)
- Large bacteria (1-10 μm, visible at 400x magnification)
- Small protozoans (10-100 μm)

Adjust the detection parameters in the code based on your specific sample and microscope configuration.

```python
import cv2
import numpy as np
from microscope_demo_client import MicroscopeDemo
from my_secrets import (
    HIVEMQ_HOST,
    HIVEMQ_PASSWORD,
    HIVEMQ_PORT,
    HIVEMQ_USERNAME,
    MICROSCOPE_NAME,
)
import time
from PIL import Image
import io

microscope = MicroscopeDemo(
    HIVEMQ_HOST, HIVEMQ_PORT, HIVEMQ_USERNAME, HIVEMQ_PASSWORD, MICROSCOPE_NAME
)


def preprocess_image(image):
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    blurred = cv2.GaussianBlur(gray, (5, 5), 0)
    thresh = cv2.adaptiveThreshold(
        blurred, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 11, 2
    )
    return thresh


def detect_particles(image):
    processed = preprocess_image(image)
    contours, _ = cv2.findContours(
        processed, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
    )
    particles = []
    for contour in contours:
        area = cv2.contourArea(contour)
        if 10 < area < 1000:  # Adjust based on your sample and magnification
            particles.append(contour)
    return particles


def analyze_growth(previous_count, current_count):
    growth_rate = (
        (current_count - previous_count) / previous_count * 100
        if previous_count > 0
        else 0
    )
    return growth_rate


def microscope_scan_and_analyze():
    print("Starting area scan...")
    scan_images = microscope.scan([2000, 2000], [-2000, -2000])

    total_particles = 0
    for i, img_data in enumerate(scan_images):
        img = Image.open(io.BytesIO(img_data))
        img_cv = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)

        particles = detect_particles(img_cv)
        total_particles += len(particles)

        img_with_particles = img_cv.copy()
        cv2.drawContours(img_with_particles, particles, -1, (0, 255, 0), 2)

        cv2.imshow(f"Scan {i+1}/{len(scan_images)}", img_with_particles)
        cv2.waitKey(1000)

        print(
            f"Scan image {i+1}/{len(scan_images)}: Detected {len(particles)} microscopic particles"
        )

    cv2.destroyAllWindows()
    return total_particles


def main_microscope():
    try:
        previous_particle_count = 0
        while True:
            microscope.move(0, 0)
            microscope.focus()

            current_particle_count = microscope_scan_and_analyze()
            growth_rate = analyze_growth(
                previous_particle_count, current_particle_count
            )

            print(f"Total detected microscopic particles: {current_particle_count}")
            print(f"Growth rate: {growth_rate:.2f}%")

            previous_particle_count = current_particle_count
            time.sleep(3600)  # Wait for 1 hour

    except KeyboardInterrupt:
        print("Monitoring stopped")
    finally:
        microscope.end_connection()


if __name__ == "__main__":
    main_microscope()
```

Note: The visibility and detection of particles will depend on the microscope's magnification and the size of the particles you're observing. This example can be used for various types of samples, such as microspheres (1-100 μm), pollen grains (10-100 μm), large bacteria (1-10 μm, visible at 400x magnification), or small protozoans (10-100 μm).

Considering of more accurate camera calibration is crucial for precise AprilTag detection and spatial referencing, we provide the following options:

1. Default Calibration: For quick start, you can use these default parameters for the Raspberry Pi Camera Module:
   ```python
   camera_params = [1000, 1000, 320, 240]  # fx, fy, cx, cy
   ```

2. Simple Calibration: Run the provided `calibrate_camera.py` script, which uses AprilTags for calibration:
   ```
   python calibrate_camera.py
   ```
   This will generate a `calibration.json` file with your camera's parameters.

Optional: Include a ruler in your setup for size reference and manual verification of measurements.

For more information on camera calibration, refer to the OpenCV documentation on [Camera Calibration](https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html).

## Tips

1. Improving AprilTag Spatial Referencing Accuracy:
   - Perform precise camera calibration using the provided calibration script.
   - Capture calibration images from various angles and distances for better results.
   - Use larger AprilTags to improve detection accuracy at greater distances.
   - Ensure consistent and adequate lighting conditions in your working environment.

2. Enhancing Particle Detection in Microscope Automation:
   - Fine-tune the threshold parameters in the `detect_particles` function for your specific samples.
   - Experiment with advanced image processing techniques like edge detection or morphological operations.
   - Develop custom detection algorithms tailored to your particular type of sample if needed.
   - Consider using machine learning approaches for more complex particle recognition tasks.

3. Extending the System for Multi-Object Recognition and Tracking:
   - Integrate deep learning models such as YOLO or SSD for object detection and classification.
   - Combine these models with the AprilTag system for comprehensive scene understanding.
   - Implement a tracking algorithm to maintain object identity across frames.
   - Use a database to store and retrieve object information based on detected features or tags.

4. Handling Image Analysis at Different Microscope Magnification Levels:
   - Create separate sets of image processing parameters optimized for each magnification level.
   - Implement a calibration process using standard samples of known size.
   - Determine and store the pixel-to-actual size conversion ratio for each magnification level.
   - Develop a system to automatically adjust processing parameters based on the current magnification setting.


## 🚀 Quiz

::::{tab-set}
:sync-group: category

:::{tab-item} W 2024
:sync: w2024

:::

:::{tab-item} Sp/Su 2024
:sync: sp2024

https://q.utoronto.ca/courses/370069/assignments/1393649
:::

:::{tab-item} Sp/Su 2025
:sync: sp2025

https://q.utoronto.ca/courses/393350/assignments/1499711?display=full_width_with_nav
:::

::::

## 📄 Assignment

::::{tab-set}
:sync-group: category

:::{tab-item} W 2024
:sync: w2024

:::

:::{tab-item} Sp/Su 2024
:sync: sp2024

https://q.utoronto.ca/courses/370069/assignments/1394528
:::

:::{tab-item} Sp/Su 2025
:sync: sp2025

https://q.utoronto.ca/courses/393350/assignments/1499712?display=full_width_with_nav
:::

::::