FCN-语义分割【玩转华为云】

FCN-语义分割

一、模型训练与转换

FCN（全卷积网络，Fully Convolutional Networks）是用于语义分割任务的一种深度学习模型架构，引入了跳跃结构（Skip Architecture），通过融合浅层和深层的特征图，保留更多的细节信息，提升分割精度。此外，FCN还利用多尺度上下文聚合，捕捉不同层级的特征，增强了对不同大小目标的识别能力。FCN的成功推动了语义分割领域的发展，成为后续许多先进模型的基础。

模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。

在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。

二、应用开发

1. 创建工程

在ModelBox sdk目录下使用create.bat创建FCN工程：

PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t server -n FCN 
...
success: create FCN in D:\modelbox-win10-x64-1.5.3\workspace

create.bat工具的参数中，-t参数，表示所创建实例的类型，包括server（ModelBox工程）、python（Python功能单元）、c++（C++功能单元）、infer（推理功能单元）等；-n参数，表示所创建实例的名称；-s参数，表示将使用后面参数值代表的模板创建工程，而不是创建空的工程。

2. 创建推理功能单元

在ModelBox sdk目录下使用create.bat创建fcn_infer推理功能单元:

PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t infer -n fcn_infer -p FCN 
...
success: create infer fcn_infer in D:\modelbox-win10-x64-1.5.3\workspace\FCN/model/fcn_infer

create.bat工具使用时，-t infer即表示创建的是推理功能单元；-n xxx_infer表示创建的功能单元名称为xxx_infer；-p表示所创建的功能单元属于FCN应用。
下载转换好的FCN.onnx模型到FCN\model目录下，修改推理功能单元fcn_infer.toml模型的配置文件：

# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved.

[base]
name = "fcn_infer"
device = "cpu"
version = "1.0.0"
description = "your description"
entry = "./FCN.onnx"  # model file path, use relative path
type = "inference" 
virtual_type = "onnx" # inference engine type: win10 now only support onnx
group_type = "Inference"  # flowunit group attribution, do not change

# Input ports description
[input]
[input.input1]  # input port number, Format is input.input[N]
name = "Input"  # input port name
type = "float"  # input port data type ,e.g. float or uint8
device = "cpu"  # input buffer type: cpu, win10 now copy input from cpu

# Output ports description
[output]
[output.output1] # output port number, Format is output.output[N]
name = "Output"  # output port name
type = "float"   # output port data type ,e.g. float or uint8

3. 创建后处理功能单元

在ModelBox sdk目录下使用create.bat创建fcn_post后处理功能单元:

PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t python -n fcn_post -p FCN
...
success: create python fcn_post in D:\modelbox-win10-x64-1.5.3\workspace\FCN/etc/flowunit/fcn_post

create.bat工具使用时，-t python即表示创建的是通用功能单元；-n xxx_post表示创建的功能单元名称为xxx_post；-p表示所创建的功能单元属于FCN应用。

a. 修改配置文件

我们的模型有一个输入和输出，对116种果蔬病虫害进行分割，加上背景总共是117类：

# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved.

# Basic config
[base]
name = "fcn_post" # The FlowUnit name
device = "cpu" # The flowunit runs on cpu
version = "1.0.0" # The version of the flowunit
type = "python" # Fixed value, do not change
description = "description" # The description of the flowunit
entry = "fcn_post@fcn_postFlowUnit" # Python flowunit entry function
group_type = "Generic"  # flowunit group attribution, change as Input/Output/Image/Generic ...

# Flowunit Type
stream = false # Whether the flowunit is a stream flowunit
condition = false # Whether the flowunit is a condition flowunit
collapse = false # Whether the flowunit is a collapse flowunit
collapse_all = false # Whether the flowunit will collapse all the data
expand = false #  Whether the flowunit is a expand flowunit

# The default Flowunit config
[config]
num_classes = 117
net_w = 224
net_h = 224

# Input ports description
[input]
[input.input1] # Input port number, the format is input.input[N]
name = "in_image" # Input port name
type = "uint8" # Input port type

[input.input2] # Input port number, the format is input.input[N]
name = "in_feat" # Input port name
type = "float" # Input port type

# Output ports description
[output]
[output.output1] # Output port number, the format is output.output[N]
name = "out_image" # Output port name
type = "uint8" # Output port type

b. 修改逻辑代码

# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import _flowunit as modelbox
import numpy as np
import cv2

class fcn_postFlowUnit(modelbox.FlowUnit):
    # Derived from modelbox.FlowUnit
    def __init__(self):
        super().__init__()

    def open(self, config):
        # Open the flowunit to obtain configuration information
        self.params = {}
        self.params['num_classes'] = config.get_int('num_classes')
        self.params['net_w'] = config.get_int('net_w')
        self.params['net_h'] = config.get_int('net_h')

        return modelbox.Status.StatusCode.STATUS_SUCCESS

    def process(self, data_context):
        # Process the data
        in_image = data_context.input("in_image")
        in_feat = data_context.input("in_feat")
        out_image = data_context.output("out_image")

        # fcn_post process code.
        # Remove the following code and add your own code here.
        for buffer_image, buffer_feat in zip(in_image, in_feat):
            channel = buffer_image.get('channel')
            width = buffer_image.get('width')
            height = buffer_image.get('height')

            image = np.array(buffer_image.as_object(), dtype=np.uint8, copy=False)
            image = image.reshape(height, width, channel)

            feat = np.array(buffer_feat.as_object(), dtype=np.float32, copy=False)
            feat = feat.reshape(self.params['net_h'], self.params['net_w'], self.params['num_classes'])

            mask = np.argmax(feat, axis=-1).astype(np.uint8)
            mask = cv2.resize(mask, (width, height), interpolation=cv2.INTER_NEAREST)

            overlay = np.zeros_like(image)
            for i in range(1, self.params['num_classes']):
                color = np.random.randint(0, 255, (3,)).tolist()
                overlay[mask==i] = color

            result_image = cv2.addWeighted(image[..., ::-1], 0.5, overlay, 0.5, 0)
            add_buffer = modelbox.Buffer(self.get_bind_device(), result_image)
            add_buffer.copy_meta(buffer_image)
            out_image.push_back(add_buffer)

        return modelbox.Status.StatusCode.STATUS_SUCCESS

    def close(self):
        # Close the flowunit
        return modelbox.Status()

    def data_pre(self, data_context):
        # Before streaming data starts
        return modelbox.Status()

    def data_post(self, data_context):
        # After streaming data ends
        return modelbox.Status()

    def data_group_pre(self, data_context):
        # Before all streaming data starts
        return modelbox.Status()

    def data_group_post(self, data_context):
        # After all streaming data ends
        return modelbox.Status()

4. 修改应用的流程图

FCN工程graph目录下存放流程图，默认的流程图FCN.toml与工程同名：

# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved.

[driver]
dir = ["${HILENS_APP_ROOT}/etc/flowunit",
"${HILENS_APP_ROOT}/etc/flowunit/cpp",
"${HILENS_APP_ROOT}/model",
"${HILENS_MB_SDK_PATH}/flowunit"]
skip-default = true
[profile]
profile=false
trace=false
dir="${HILENS_DATA_DIR}/mb_profile"
[graph]
format = "graphviz"
graphconf = """digraph FCN {
    node [shape=Mrecord]
    queue_size = 4
    batch_size = 1
    input1[type=input,flowunit=input,device=cpu,deviceid=0]
    httpserver_sync_receive[type=flowunit, flowunit=httpserver_sync_receive_v2, device=cpu, deviceid=0, time_out_ms=5000, endpoint="http://0.0.0.0:1234/v1/FCN", max_requests=100]
    image_decoder[type=flowunit, flowunit=image_decoder, device=cpu, key="image_base64", queue_size=4]
    image_resize[type=flowunit, flowunit=resize, device=cpu, deviceid=0, image_width=224, image_height=224]
    normalize[type=flowunit, flowunit=normalize, device=cpu, deviceid=0, standard_deviation_inverse="0.003921568627450,0.003921568627450,0.003921568627450"]
    fcn_infer[type=flowunit, flowunit=fcn_infer, device=cpu, deviceid=0, batch_size=1]
    fcn_post[type=flowunit, flowunit=fcn_post, device=cpu, deviceid=0]
    httpserver_sync_reply[type=flowunit, flowunit=httpserver_sync_reply_v2, device=cpu, deviceid=0]
    
    input1:input -> httpserver_sync_receive:in_url
    httpserver_sync_receive:out_request_info -> image_decoder:in_encoded_image
    image_decoder:out_image -> image_resize:in_image
    image_resize:out_image -> normalize:in_data
    normalize:out_data -> fcn_infer:Input
    image_decoder:out_image -> fcn_post:in_image
    fcn_infer:Output -> fcn_post:in_feat
    fcn_post:out_image -> httpserver_sync_reply:in_reply_info
}"""
[flow]
desc = "FCN run in modelbox-win10-x64"

在命令行中运行.\create.bat -t editor即可打开ModelBox图编排界面，可以实时修改并查看项目的流程图：

PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t editor

5. 运行应用

在FCN工程目录下执行.\bin\main.bat运行应用：

PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\FCN
PS D:\modelbox-win10-x64-1.5.3\workspace\FCN> .\bin\main.bat

在FCN工程data目录下新建test_http.py测试脚本：

import cv2
import json
import base64
import requests
import numpy as np
if __name__ == "__main__":
    port = 1234
    ip = "127.0.0.1"
    url = "/v1/FCN"

    img_path = "apple_black_rot_google_0056.jpg"
    img_data = cv2.imread(img_path)
    img_data = cv2.cvtColor(img_data, cv2.COLOR_BGR2RGB)
    img_str = cv2.imencode('.jpg', img_data)[1].tobytes()
    img = base64.b64encode(img_str)
    img_base64_str = str(img, encoding='utf8')
    params = {"image_base64": img_base64_str}
    response = requests.post(f'http://{ip}:{port}{url}', data=json.dumps(params),
                             headers={"Content-Type": "application/json"})
    h, w, c = img_data.shape
    img_array = np.frombuffer(response.content, np.uint8)
    img_array = img_array.reshape((h, -1, c))
    cv2.imwrite("res.jpg", img_array)

在FCN工程data目录下存放测试图片：

在另一个终端中进入FCN工程目录data文件夹下：

PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\FCN\data

首先安装requests依赖包：

PS D:\modelbox-win10-x64-1.5.3\workspace\FCN\data> D:\modelbox-win10-x64-1.5.3\python-embed\python.exe -m pip install requests

然后运行test_http.py脚本发起HTTP请求测试：

PS D:\modelbox-win10-x64-1.5.3\workspace\FCN\data> D:\modelbox-win10-x64-1.5.3\python-embed\python.exe .\test_http.py

测试图片的分割结果res.jpg将保存在FCN工程data目录下：

三、小结

本节介绍了如何使用ModelArts和ModelBox训练开发一个FCN果蔬病虫害分割的AI应用，我们只需要准备模型文件以及简单的配置即可创建一个HTTP服务。同时我们可以了解到FCN网络的基本结构、数据处理和模型训练方法，以及对应推理应用的逻辑。