Image processing is a crucial part of various applications, from computer vision to machine learning. One fundamental technique in image processing is edge detection and the Sobel filter is a popular method for doing so. This blog post will guide you through creating a Sobel filter IP core using High-Level-Synthesis (HLS), integrating it into Vivado Design Suite, and controlling it with a PYNQ Jupyter Notebook on a PYNQ Z2 board.
Step 1: Creating the Sobel filter in HLS
Vitis HLS
HLS allows you to design hardware using high-level programming with a decent amount of optimization. The programming language is similar to C or C++. We will start by creating a Sobel filter in HLS.
#include "hls_stream.h"
#include "ap_int.h"
#include "ap_axi_sdata.h"
typedef ap_axis<32, 2, 5, 6> pixel_t;
void sobel_filter(hls::stream<pixel_t> &input, hls::stream<pixel_t> &output, int rows, int cols) {
#pragma HLS INTERFACE axis port=input
#pragma HLS INTERFACE axis port=output
#pragma HLS INTERFACE s_axilite port=rows
#pragma HLS INTERFACE s_axilite port=cols
#pragma HLS INTERFACE s_axilite port=return
int Gx[3][3] = { {-1, 0, 1}, {-2, 0, 2}, {-1, 0, 1} };
int Gy[3][3] = { { 1, 2, 1}, { 0, 0, 0}, {-1, -2, -1} };
pixel_t window[3][3];
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
#pragma HLS PIPELINE
pixel_t pixel_in = input.read();
// Update window
for (int k = 0; k < 2; k++) {
for (int l = 0; l < 3; l++) {
window[k][l] = window[k+1][l];
}
}
window[2][0] = window[2][1];
window[2][1] = window[2][2];
window[2][2] = pixel_in;
// Apply Sobel filter
int Gx_val = 0;
int Gy_val = 0;
for (int k = 0; k < 3; k++) {
for (int l = 0; l < 3; l++) {
Gx_val += window[k][l].data * Gx[k][l];
Gy_val += window[k][l].data * Gy[k][l];
}
}
// Calculate magnitude using L1 norm (|Gx| + |Gy|)
int abs_Gx_val = Gx_val < 0 ? -Gx_val : Gx_val;
int abs_Gy_val = Gy_val < 0 ? -Gy_val : Gy_val;
int magnitude = abs_Gx_val + abs_Gy_val;
pixel_t pixel_out;
pixel_out.data = magnitude > 255 ? 255 : magnitude;
pixel_out.keep = pixel_in.keep;
pixel_out.strb = pixel_in.strb;
pixel_out.user = pixel_in.user;
pixel_out.last = pixel_in.last;
pixel_out.id = pixel_in.id;
pixel_out.dest = pixel_in.dest;
output.write(pixel_out);
}
}
}
Data Types
This code is straightforward and similar to standard implementations in C++, with some differences focused on function arguments and hardware-specific pragmas.
typedef ap_axis<32, 2, 5, 6> pixel_t;This line defines a custom data type pixel_t using the ap_axis template. The ap_axis type is designed for AXI-Stream interfaces and includes additional signals for data integrity and control.
typedef ap_axis<32, 2, 5, 6>- 32: Width of the data bus in bits.
- 2: Number of user signals.
- 5: Number of ID signals.
- 6: Number of destination signals.
This custom data type allows the IP core to be connected to the DMA interface which is more complex compared to AXI Lite. That data type is the input and output stream type for data transition between IP cores.
#pragma HLS INTERFACE axis port=input
#pragma HLS INTERFACE axis port=output
#pragma HLS INTERFACE s_axilite port=rows
#pragma HLS INTERFACE s_axilite port=cols
#pragma HLS INTERFACE s_axilite port=returnThese pragmas are used by the HLS tool to generate the appropriate hardware interfaces for the function. Here are 2 types of interfaces, one for the AXI Lite bus and one for the DMA interface stream.
Step 2: Exporting and Integrating the Sobel IP Core in Vivado
In this part, we are required to set the top functions in Vitis HLS and run synthesis. The last part is export where we receive a .zip file for IP core configuration.
Once the IP core is created, the next step is to integrate it into a Vivado project.
Vivado
We will need to add the created IP core (zip file) into the IP repository inside the Vivado project. Before running into the project, below is the list of all required IP cores for this example to work.
Required IP cores in Vivado
- ZYNQ7 Processing System – used to control and configure the Sobel filter IP and manage data transfers between the PS and PL.
- AXI DMA – facilitates the transfer of image data to the Sobel filter IP core and retrieves the processed data.
- AXI Interconnect – connects the AXI DMA, ZYNQ7 Processing System, and Sobel filter IP core for memory-mapped communication.
- Sobel Filter IP Core – processes the input image data to detect edges and outputs the processed data.
Steps to Add These IP Cores in Vivado
- ZYNQ7 Processing System – configure this block to have High Processing Port HP0 and run block connection and automation.
- AXI DMA – configure it to support read & write channel operations. Set memory width to 64 bits and stream width to 32.
- AXI Interconnect – Connection automation will do everything.
- Sobel Filter IP Core – Connect the AXI4-Stream input and output interfaces to the AXI DMA and configure the control interfaces via AXI Lite.
All connected in Vivado it would look like in Picture 1. Below.
Step 3: Controlling the Sobel IP Core with PYNQ
With the hardware setup, the final step is to control the Sobel filter IP core using a Jupyter Notebook on the PYNQ Z2 board. After fully uploading the bitstream and hardware description files, we aim to create a Python code for testing the Sobel filter IP core implementation on FPGA. The code block of our approach is presented below:
from pynq import Overlay, allocate
from pynq.lib.dma import DMA
import numpy as np
import cv2
from matplotlib import pyplot as plt
import time
# Define Sobel filter register addresses
XSOBEL_FILTER_CONTROL_ADDR_AP_CTRL = 0x00
XSOBEL_FILTER_CONTROL_ADDR_GIE = 0x04
XSOBEL_FILTER_CONTROL_ADDR_IER = 0x08
XSOBEL_FILTER_CONTROL_ADDR_ISR = 0x0c
XSOBEL_FILTER_CONTROL_ADDR_ROWS_DATA = 0x10
XSOBEL_FILTER_CONTROL_ADDR_COLS_DATA = 0x18
# Load the overlay
overlay = Overlay("design_1.bit")
# Initialize DMA
dma = overlay.axi_dma_0
# Reference to the Sobel filter IP core
sobel_filter_ip = overlay.sobel_filter_0
# Function to read and print IP core registers
def print_ip_registers(ip, num_registers):
print(f"Reading {num_registers} registers of the IP core...")
for reg_offset in range(0, num_registers * 4, 4):
reg_value = ip.read(reg_offset)
print(f"Register 0x{reg_offset:02X}: 0x{reg_value:08X}")
# Define the number of registers to read
num_registers = 10 # Adjust as per your IP core's documentation
# Print the registers of the Sobel filter IP core
print_ip_registers(sobel_filter_ip, num_registers)
# Load an image using OpenCV
image = cv2.imread('lena.jpg', cv2.IMREAD_GRAYSCALE)
rows, cols = image.shape
# Ensure image is uint32
image = image.astype(np.uint32)
# Allocate buffers for input and output
in_buffer = allocate(shape=(rows*cols,), dtype=np.uint32)
out_buffer = allocate(shape=(rows*cols,), dtype=np.uint32)
# Copy the image data to the input buffer
np.copyto(in_buffer, image.flatten())
# Set the parameters for the Sobel filter IP
sobel_filter_ip.write(XSOBEL_FILTER_CONTROL_ADDR_ROWS_DATA, rows)
sobel_filter_ip.write(XSOBEL_FILTER_CONTROL_ADDR_COLS_DATA, cols)
# Print registers after setting rows and cols
print("Registers after setting rows and cols:")
print_ip_registers(sobel_filter_ip, num_registers)
# Start the Sobel filter IP core
sobel_filter_ip.write(XSOBEL_FILTER_CONTROL_ADDR_AP_CTRL, 0x81) # Start the IP core
# Transfer input data to FPGA
dma.sendchannel.transfer(in_buffer)
# Transfer output data from FPGA
dma.recvchannel.transfer(out_buffer)
# Wait for sendchannel to complete
timeout = 5 # seconds
start_time = time.time()
while not dma.sendchannel.idle:
if time.time() - start_time > timeout:
print("Timeout occurred during dma.sendchannel.wait()")
break
# Wait for recvchannel to complete
start_time = time.time()
while not dma.recvchannel.idle:
if time.time() - start_time > timeout:
print("Timeout occurred during dma.recvchannel.wait()")
break
# Wait for the Sobel filter IP core to complete
start_time = time.time()
while not sobel_filter_ip.read(XSOBEL_FILTER_CONTROL_ADDR_AP_CTRL) & 0x02:
if time.time() - start_time > timeout:
print("Timeout occurred during Sobel filter IP core processing")
break
# Print registers after starting the IP core
print("Registers after starting the IP core:")
print_ip_registers(sobel_filter_ip, num_registers)
# Reshape the output buffer to the image shape
output_image = out_buffer.reshape((rows, cols)).astype(np.uint8)
# Display the original and processed images
plt.figure(figsize=(10,5))
plt.subplot(1, 2, 1)
plt.title("Original Image")
plt.imshow(image, cmap='gray')
plt.subplot(1, 2, 2)
plt.title("Sobel Filtered Image")
plt.imshow(output_image, cmap='gray')
plt.show()
The final result of the running Sobel filter on PYNQ Z2 is shown in Picture 2. below.
Conclusion
By following the steps outlined in this blog post, you should now have a working Sobel filter implemented as an IP core, integrated into a Vivado project, and controlled via a PYNQ Jupyter Notebook. This project demonstrates the power of HLS for designing custom hardware accelerators and the ease of using PYNQ for rapid prototyping and testing.
