ModelBox猫狗识别【玩转华为云】
【摘要】 本文主要进行算法的验证和部署，使用ModelBox开发AI应用，实现了猫狗图像的分类以及头部的定位，同时输出mask掩模图。目前该算法仅支持单个目标的检测和分割，后续会扩展到多个目标。
TensorFlow猫狗识别Oxford-IIIT图像定位与分割算法验证及部署
本文主要进行算法验证和部署测试，该算法实现了猫狗图像的分类以及头部的定位，同时输出mask掩模图。目前仅支持单个目标的检测和分割，后续会扩展到多个目标。之后使用ModelBox进行AI应用开发，模型最终运行效果如下：
本案例所需资源（代码、模型、测试数据等）均可从网盘链接下载。
由于笔者晚上睡觉时把电脑压坏了，于是趁着618活动之际入手了一台matebook16，笔者会在上面部署模型，实际检验电脑的性能。
模型训练
代码实现可以参考我发布的Notebook，可以点击Run in ModelArts一键运行：
导出的onnx模型结构如下所示：
ModelBox推理真的高效吗
我们分别使用onnxruntime与ModelBox Windows SDK对相同的模型实现相同的推理进行端到端的性能对比，我们准备了(1080p, 25fps)的视频作为测试输入。
原生onnxruntime推理：
视频检测的帧率只有4：
原生API推理代码位于资源包的onnxruntime_infer目录下：
"""
OpenCV 读取摄像头视频视频流，使用原生的onnxruntime推理
"""
# 导入OpenCV
import cv2
import time
import drawUtils
import numpy as np
import onnxruntime
from PIL import Image
# cap = cv2.VideoCapture(0)
cap = cv2.VideoCapture('video/1686284611402.mp4')
# cap = cv2.VideoCapture('video/1686307053582.mp4')
if not cap.isOpened():
print('文件不存在或编码错误')
else:
i = 0
fps = 25
start_time = time.time()
font = cv2.FONT_HERSHEY_PLAIN
index_to_clss = {0: 'cat', 1: 'dog'}
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
onnx_model = onnxruntime.InferenceSession('model/oxford.onnx')
writer = cv2.VideoWriter('VideoOut.mp4',cv2.VideoWriter_fourcc(*'X264'),fps,(width,height))
while cap.isOpened():
ret,frame = cap.read()
if ret:
img = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
img = cv2.resize(img, (224, 224))
img = img[...,::-1]/255.
img = img.astype(np.float32)
data = np.expand_dims(img, axis=0)
onnx_input ={onnx_model.get_inputs()[0].name: data}
pred_mask, clss, out1, out2, out3, out4 = onnx_model.run(None, onnx_input)
pred_mask = np.argmax(pred_mask, axis=-1)
pred_mask = pred_mask.astype(np.float32)
pred_mask = cv2.resize(pred_mask[0], (width, height))
pred_mask = pred_mask.astype(np.uint8)
mask = np.zeros(pred_mask.shape)
mask_0 = np.where(pred_mask==0, 255, mask)
mask_2 = np.where(pred_mask==2, 255, mask)
mask_r = Image.fromarray(mask_0).convert('L')
mask_g = Image.fromarray(mask_2).convert('L')
mask_b = Image.fromarray(mask_0).convert('L')
img_rgb = Image.merge('RGB', (mask_r, mask_g, mask_b))
img_rgb = np.asarray(img_rgb)
img_add = cv2.addWeighted(frame, 1, img_rgb, 0.5, 0)
if clss[0]>0.99 or clss[0]<0.1:
xmin, ymin, xmax, ymax = int(out1[0]*width), int(out2[0]*height), int(out3[0]*width), int(out4[0]*height)
img_add = cv2.rectangle(img=img_add,pt1=(xmin,ymin),pt2=(xmax,ymax),color=(255,0,0),thickness=10)
img_add = cv2.putText(img=img_add,text=index_to_clss.get((clss[0] > 0.5).astype('int')[0]),org=(xmin,ymin),fontFace=font,fontScale=5,color=(0,0,255),thickness=5,lineType=cv2.LINE_AA)
# 计算FPS
i += 1
now = time.time()
fps_text = int(1 / ( now - start_time))
start_time = now
print('oxford post ' + str(i))
# 添加中文（首先导入模块）
img_add = drawUtils.cv2AddChineseText(img_add, '帧率：'+str(fps_text), (20,50), textColor=(0, 255, 0), textSize=30)
# 显示画面
# cv2.imshow('oxford',img_add)
writer.write(img_add)
# 退出条件
if cv2.waitKey(1) & 0xFF == ord('q'):
break
else:
break
cap.release()
cv2.destroyAllWindows()
ModelBox应用开发
使用VS Code连接到ModelBox SDK所在目录或者远程开发板进行应用开发，从零开发工程。
创建工程
在SDK目录下使用create.py脚本创建工程，命名为oxford工程。
yanso@hou MINGW64 /d/modelbox-win10-x64-1.5.3
$ python ./create.py -t server -n oxford
sdk version is modelbox-win10-x64-1.5.3
dos2unix: converting file D:\modelbox-win10-x64-1.5.3\workspace\oxford/graph\modelbox.conf to Unix format...
dos2unix: converting file D:\modelbox-win10-x64-1.5.3\workspace\oxford/graph\oxford.toml to Unix format...
dos2unix: converting file D:\modelbox-win10-x64-1.5.3\workspace\oxford/bin\mock_task.toml to Unix format...
success: create oxford in D:\modelbox-win10-x64-1.5.3\workspace
创建推理功能单元
yanso@hou MINGW64 /d/modelbox-win10-x64-1.5.3
$ python ./create.py -t infer -n oxford_infer -p oxford
sdk version is modelbox-win10-x64-1.5.3
success: create infer oxford_infer in D:\modelbox-win10-x64-1.5.3\workspace\oxford/model/oxford_infer
可以看到创建好的推理功能单元在项目工程的model目录下面：
将我们转换好的oxford.onnx模型拖到oxford_infer目录下，接着编辑.toml文件，主要修改模型路径与输入输出，由于我们的模型有一个来自cpu的float类型输入与六个float类型的输出，所以对配置文件编辑如下：
# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved.
[base]
name = "oxford_infer"
device = "cpu"
version = "1.0.0"
description = "your description"
entry = "./oxford.onnx" # model file path, use relative path
type = "inference"
virtual_type = "onnx" # inference engine type: win10 now only support onnx
group_type = "Inference" # flowunit group attribution, do not change
# Input ports description
[input]
[input.input1] # input port number, Format is input.input[N]
name = "Input" # input port name
type = "float" # input port data type ,e.g. float or uint8
device = "cpu" # input buffer type: cpu, win10 now copy input from cpu
# Output ports description
[output]
[output.output1] # output port number, Format is output.output[N]
name = "Output1" # output port name
type = "float" # output port data type ,e.g. float or uint8
[output.output2] # output port number, Format is output.output[N]
name = "Output2" # output port name
type = "float" # output port data type ,e.g. float or uint8
[output.output3] # output port number, Format is output.output[N]
name = "Output3" # output port name
type = "float" # output port data type ,e.g. float or uint8
[output.output4] # output port number, Format is output.output[N]
name = "Output4" # output port name
type = "float" # output port data type ,e.g. float or uint8
[output.output5] # output port number, Format is output.output[N]
name = "Output5" # output port name
type = "float" # output port data type ,e.g. float or uint8
[output.output6] # output port number, Format is output.output[N]
name = "Output6" # output port name
type = "float" # output port data type ,e.g. float or uint8
创建后处理功能单元
yanso@hou MINGW64 /d/modelbox-win10-x64-1.5.3
$ python ./create.py -t python -n oxford_post -p oxford
sdk version is modelbox-win10-x64-1.5.3
success: create python oxford_post in D:\modelbox-win10-x64-1.5.3\workspace\oxford/etc/flowunit/oxford_post
后处理功能单元主要对模型推理结果进行解码，可以看到在项目工程的etc/flowunit目录下面已经生成了 oxford_post功能单元，包含.toml配置文件与.py功能代码文件：
对于后处理功能单元的配置文件，我们在config中配置参数，接收六个float类型的推理结果与一个uint8类型的原图，输出和原图相同大小的掩模图以及图像类别和检测框。
# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved.
# Basic config
[base]
name = "oxford_post" # The FlowUnit name
device = "cpu" # The flowunit runs on cpu
version = "1.0.0" # The version of the flowunit
type = "python" # Fixed value, do not change
description = "description" # The description of the flowunit
entry = "oxford_post@oxford_postFlowUnit" # Python flowunit entry function
group_type = "Generic" # flowunit group attribution, change as Input/Output/Image/Generic ...
# Flowunit Type
stream = false # Whether the flowunit is a stream flowunit
condition = false # Whether the flowunit is a condition flowunit
collapse = false # Whether the flowunit is a collapse flowunit
collapse_all = false # Whether the flowunit will collapse all the data
expand = false # Whether the flowunit is a expand flowunit
# The default Flowunit config
[config]
mask_h = 224
mask_w = 224
# Input ports description
[input]
[input.input1] # Input port number, the format is input.input[N]
name = "in_feat1" # Input port name
type = "float" # Input port type
[input.input2] # Input port number, the format is input.input[N]
name = "in_feat2" # Input port name
type = "float" # Input port type
[input.input3] # Input port number, the format is input.input[N]
name = "in_feat3" # Input port name
type = "float" # Input port type
[input.input4] # Input port number, the format is input.input[N]
name = "in_feat4" # Input port name
type = "float" # Input port type
[input.input5] # Input port number, the format is input.input[N]
name = "in_feat5" # Input port name
type = "float" # Input port type
[input.input6] # Input port number, the format is input.input[N]
name = "in_feat6" # Input port name
type = "float" # Input port type
[input.input7] # Input port number, the format is input.input[N]
name = "in_image" # Input port name
type = "uint8" # Input port type
# Output ports description
[output]
[output.output1] # Output port number, the format is output.output[N]
name = "out_mask" # Output port name 掩模图 + 图像类别检测框（以属性方式附加在图像上）
type = "uint8" # Output port type
接下来补充后处理的逻辑代码：
# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import _flowunit as modelbox
import numpy as np
import cv2
class oxford_postFlowUnit(modelbox.FlowUnit):
# Derived from modelbox.FlowUnit
def __init__(self):
super().__init__()
def open(self, config):
# Open the flowunit to obtain configuration information
self.mask_h = config.get_int('mask_h', 224)
self.mask_w = config.get_int('mask_w', 224)
return modelbox.Status.StatusCode.STATUS_SUCCESS
def process(self, data_context):
# Process the data 从Data_Context中获取输入输出的BufferList对象
in_mask = data_context.input("in_feat1")
in_clss = data_context.input("in_feat2")
in_out1 = data_context.input("in_feat3")
in_out2 = data_context.input("in_feat4")
in_out3 = data_context.input("in_feat5")
in_out4 = data_context.input("in_feat6")
in_image = data_context.input("in_image")
out_mask = data_context.output("out_mask")
# oxford_post process code.
# Remove the following code and add your own code here.
for buffer_mask, buffer_clss, buffer_out1, buffer_out2, buffer_out3, buffer_out4, buffer_image in zip(in_mask, in_clss, in_out1, in_out2, in_out3, in_out4, in_image):
# 获取输入Buffer的属性信息
width = buffer_image.get('width')
height = buffer_image.get('height')
channel = buffer_image.get('channel')
# 将输入Buffer转换为numpy对象
buffer_mask = np.array(buffer_mask.as_object(), copy=False)
buffer_mask = buffer_mask.reshape((self.mask_h, self.mask_w, -1))
buffer_mask = np.argmax(buffer_mask, axis=-1)
buffer_mask = buffer_mask.astype(np.float32)
buffer_mask = cv2.resize(buffer_mask, (width, height))
buffer_mask = buffer_mask.astype(np.uint8)
buffer_clss = np.array(buffer_clss.as_object(), copy=False)
buffer_out1 = np.array(buffer_out1.as_object(), copy=False)
buffer_out2 = np.array(buffer_out2.as_object(), copy=False)
buffer_out3 = np.array(buffer_out3.as_object(), copy=False)
buffer_out4 = np.array(buffer_out4.as_object(), copy=False)
# 解码出图像类别检测框
class_bboxes = self.decode_class_bboxes(buffer_clss, buffer_out1, buffer_out2, buffer_out3, buffer_out4, width, height)
# 将业务处理返回的结果数据转换为Buffer
out_buffer = modelbox.Buffer(self.get_bind_device(), buffer_mask)
# 设置输出Buffer的Meta信息，此处拷贝输入Buffer的部分Meta信息，其余差异的部分再进行设置
out_buffer.copy_meta(buffer_image)
# 将图像类别检测框作为属性附在输出Buffer上
out_buffer.set("class_bboxes", class_bboxes)
# 将输出Buffer放入输出BufferList中
out_mask.push_back(out_buffer)
return modelbox.Status.StatusCode.STATUS_SUCCESS
def decode_class_bboxes(self, buffer_clss, buffer_out1, buffer_out2, buffer_out3, buffer_out4, width, height):
# 原始检测框数据归一化到[0,1]，此处需还原到原图中的坐标
clss = float(buffer_clss)
xmin = int(buffer_out1 * width)
ymin = int(buffer_out2 * height)
xmax = int(buffer_out3 * width)
ymax = int(buffer_out4 * height)
class_bboxes = [clss, xmin, ymin, xmax, ymax]
return class_bboxes
def close(self):
# Close the flowunit
return modelbox.Status()
def data_pre(self, data_context):
# Before streaming data starts
return modelbox.Status()
def data_post(self, data_context):
# After streaming data ends
return modelbox.Status()
def data_group_pre(self, data_context):
# Before all streaming data starts
return modelbox.Status()
def data_group_post(self, data_context):
# After all streaming data ends
return modelbox.Status()
创建绘图功能单元
依然是万能的create.py脚本：
yanso@hou MINGW64 /d/modelbox-win10-x64-1.5.3
$ python ./create.py -t python -n oxford_draw -p oxford
sdk version is modelbox-win10-x64-1.5.3
success: create python oxford_draw in D:\modelbox-win10-x64-1.5.3\workspace\oxford/etc/flowunit/oxford_draw
可以看到在项目工程的etc/flowunit目录下面已经生成了两个功能单元：
对于绘图功能单元，我们需要将后处理功能单元解码后的结果和原图进行合成，生成新的图像，因此有两个输入和一个输出，修改该功能单元的配置文件：
# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved.
# Basic config
[base]
name = "oxford_draw" # The FlowUnit name
device = "cpu" # The flowunit runs on cpu
version = "1.0.0" # The version of the flowunit
type = "python" # Fixed value, do not change
description = "description" # The description of the flowunit
entry = "oxford_draw@oxford_drawFlowUnit" # Python flowunit entry function
group_type = "Generic" # flowunit group attribution, change as Input/Output/Image/Generic ...
# Flowunit Type
stream = false # Whether the flowunit is a stream flowunit
condition = false # Whether the flowunit is a condition flowunit
collapse = false # Whether the flowunit is a collapse flowunit
collapse_all = false # Whether the flowunit will collapse all the data
expand = false # Whether the flowunit is a expand flowunit
# The default Flowunit config
[config]
item = "value"
# Input ports description
[input]
[input.input1] # Input port number, the format is input.input[N]
name = "in_mask" # Input port name
type = "uint8" # Input port type
[input.input2] # Input port number, the format is input.input[N]
name = "in_image" # Input port name
type = "uint8" # Input port type
# Output ports description
[output]
[output.output1] # Output port number, the format is output.output[N]
name = "out_image" # Output port name
type = "uint8" # Output port type
绘图代码如下：
# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import _flowunit as modelbox
from PIL import Image
import numpy as np
import cv2
class oxford_drawFlowUnit(modelbox.FlowUnit):
# Derived from modelbox.FlowUnit
def __init__(self):
super().__init__()
def open(self, config):
self.font = cv2.FONT_HERSHEY_PLAIN
self.index_to_clss = {0: 'cat', 1: 'dog'}
# Open the flowunit to obtain configuration information
return modelbox.Status.StatusCode.STATUS_SUCCESS
def process(self, data_context):
# Process the data
# 从DataContext中获取输入输出BufferList对象
in_mask = data_context.input("in_mask")
in_image = data_context.input("in_image")
out_image = data_context.output("out_image")
# oxford_draw process code.
# Remove the following code and add your own code here.
# 循环处理每一个输入Buffer数据（实际上条件功能单元的batch size为1，此处循环中只有1条数据）
for buffer_mask, buffer_image in zip(in_mask, in_image):
# 获取输入Buffer的属性信息
width = buffer_image.get('width')
height = buffer_image.get('height')
channel = buffer_image.get('channel')
# 将输入Buffer转换为numpy对象
mask_data = np.array(buffer_mask.as_object(), dtype=np.uint8, copy=False)
mask_data = mask_data.reshape(height, width)
image_data = np.array(buffer_image.as_object(), dtype=np.uint8, copy=False)
image_data = image_data.reshape(height, width, channel)
class_bboxes = buffer_mask.get("class_bboxes")
class_bboxes = np.array(class_bboxes)#.reshape(-1,5)
# 绘制图像分割结果
add_data = self.draw_image_mask(mask_data, image_data)
# 绘制图像检测结果
add_buffer = self.draw_class_bboxes(add_data, class_bboxes)
# 将业务处理返回的结果数据转换为Buffer
add_buffer = modelbox.Buffer(self.get_bind_device(), add_buffer)
# 设置输出Buffer的Meta信息，此处直接拷贝输入Buffer的Meta信息
add_buffer.copy_meta(buffer_image)
# 将输出Buffer放入输出BufferList中
out_image.push_back(add_buffer)
return modelbox.Status.StatusCode.STATUS_SUCCESS
def draw_image_mask(self, mask_data, image_data):
mask = np.zeros(mask_data.shape)
mask_0 = np.where(mask_data==0, 255, mask)
mask_2 = np.where(mask_data==2, 255, mask)
mask_r = Image.fromarray(mask_0).convert('L')
mask_g = Image.fromarray(mask_2).convert('L')
mask_b = Image.fromarray(mask_0).convert('L')
mask_rgb = Image.merge('RGB', (mask_r, mask_g, mask_b))
mask_rgb = np.asarray(mask_rgb)
add_data = cv2.addWeighted(image_data, 1, mask_rgb, 0.5, 0)
return add_data
def draw_class_bboxes(self, add_data, class_bboxes):
clss, xmin, ymin, xmax, ymax = class_bboxes
xmin = int(xmin)
ymin = int(ymin)
xmax = int(xmax)
ymax = int(ymax)
score = float(clss)
if score>0.99 or score<0.1:
add_buffer = cv2.rectangle(img=add_data,pt1=(xmin,ymin),pt2=(xmax,ymax),color=(255,0,0),thickness=10)
add_buffer = cv2.putText(img=add_buffer,text=self.index_to_clss.get((clss > 0.5).astype('int')),org=(xmin,ymin),\
fontFace=self.font,fontScale=5,color=(0,0,255),thickness=5,lineType=cv2.LINE_AA)
else:
add_buffer = add_data
return add_buffer
def close(self):
# Close the flowunit
return modelbox.Status()
def data_pre(self, data_context):
# Before streaming data starts
return modelbox.Status()
def data_post(self, data_context):
# After streaming data ends
return modelbox.Status()
def data_group_pre(self, data_context):
# Before all streaming data starts
return modelbox.Status()
def data_group_post(self, data_context):
# After all streaming data ends
return modelbox.Status()
至此，我们所有的功能单元代码都准备好了。
搭建流程图
准备好了每个功能单元，接下来要在流程图中将功能单元连接起来。创建工程时默认生成的流程图在工程目录graph文件夹下：
修改oxford.toml内容：
# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved.
[driver]
dir = ["${HILENS_APP_ROOT}/etc/flowunit",
"${HILENS_APP_ROOT}/etc/flowunit/cpp",
"${HILENS_APP_ROOT}/model",
"${HILENS_MB_SDK_PATH}/flowunit"]
skip-default = true
[profile]
profile=true
trace=true
dir="${HILENS_DATA_DIR}/mb_profile"
[graph]
format = "graphviz"
graphconf = """digraph oxford {
node [shape=Mrecord]
queue_size = 1
batch_size = 1
input1[type=input,flowunit=input,device=cpu,deviceid=0]
data_source_parser[type=flowunit, flowunit=data_source_parser, device=cpu, deviceid=0]
video_demuxer[type=flowunit, flowunit=video_demuxer, device=cpu, deviceid=0]
video_decoder[type=flowunit, flowunit=video_decoder, device=cpu, deviceid=0, pix_fmt=rgb]
image_resize[type=flowunit, flowunit=resize, device=cpu, deviceid=0, image_width=224, image_height=224]
normalize[type=flowunit, flowunit=normalize, device=cpu, deviceid=0, standard_deviation_inverse="0.003921568627450,0.003921568627450,0.003921568627450"]
oxford_infer[type=flowunit, flowunit=oxford_infer, device=cpu, deviceid=0, batch_size = 1]
oxford_post[type=flowunit, flowunit=oxford_post, device=cpu, deviceid=0]
oxford_draw[type=flowunit, flowunit=oxford_draw, device=cpu, deviceid=0]
video_out[type=flowunit, flowunit=video_out, device=cpu, deviceid=0]
input1:input -> data_source_parser:in_data
data_source_parser:out_video_url -> video_demuxer:in_video_url
video_demuxer:out_video_packet -> video_decoder:in_video_packet
video_decoder:out_video_frame -> image_resize:in_image
image_resize:out_image -> normalize:in_data
normalize:out_data -> oxford_infer:Input
oxford_infer:Output1 -> oxford_post:in_feat1
oxford_infer:Output2 -> oxford_post:in_feat2
oxford_infer:Output3 -> oxford_post:in_feat3
oxford_infer:Output4 -> oxford_post:in_feat4
oxford_infer:Output5 -> oxford_post:in_feat5
oxford_infer:Output6 -> oxford_post:in_feat6
video_decoder:out_video_frame -> oxford_post:in_image
oxford_post:out_mask -> oxford_draw:in_mask
video_decoder:out_video_frame -> oxford_draw:in_image
oxford_draw:out_image -> video_out:in_video_frame
}"""
[flow]
desc = "oxford run in modelbox-win10-x64"
其中，profile字段设为true启用性能统计功能，毕竟我们要检验一下使用ModelBox推理是否真的高效。性能统计具体加载方式可以参考ModelBox文档相关章节。
运行应用
应用的输入和输出可以在项目工程的bin/mock_task.toml中进行配置：
将资源包中的测试视频拷贝到data目录下，配置应用输入输出：
# 用于本地mock文件读取任务，脚本中已经配置了IVA_SVC_CONFIG环境变量, 添加了此文件路径
########### 请确定使用linux的路径类型，比如在windows上要用 D:/xxx/xxx 不能用D:\xxx\xxx ###########
# 任务的参数为一个压缩并转义后的json字符串
# 直接写需要转义双引号， 也可以用 content_file 添加一个json文件，如果content和content_file都存在content会被覆盖
# content_file支持绝对路径或者相对路径，不支持解析环境变量(包括${HILENS_APP_ROOT}、${HILENS_DATA_DIR}等)
[common]
content = "{\"param_str\":\"string param\",\"param_int\":10,\"param_float\":10.5}"
# 任务输入配置，mock模拟目前仅支持一路rtsp或者本地url, 当前支持以下几种输入方式：
# 1. rtsp摄像头或rtsp视频流：type="rtsp", url="rtsp://xxx.xxx" (type为rtsp的时候，支持视频中断自动重连)
# 2. 设备自带摄像头或者USB摄像头：type="url"，url="摄像头编号,比如 0 或者 1 等" (需配合local_camera功能单元使用)
# 3. 本地视频文件：type="url"，url="视频文件路径" (可以是相对路径 -- 相对这个mock_task.toml文件, 也支持从环境变量${HILENS_APP_ROOT}所在目录文件输入)
# 4. http服务：type="url", url="http://xxx.xxx"(指的是任务作为http服务启动，此处需填写对外暴露的http服务地址，需配合httpserver类的功能单元使用)
# 5. 支持多输入[input] [input1] [input2] ...，对应的输出为[output] [output1] [output2] ...，如果使用videoout功能单元输出，则输入和输出个数必须匹配，同时url不能重名
[input]
type = "url"
url = "../data/1686284611402.mp4"
# 任务输出配置，当前支持以下几种输出方式：
# 1. rtsp视频流：type="local", url="rtsp://xxx.xxx"
# 2. 本地屏幕：type="local", url="0:xxx" (设备需要接显示器，系统需要安装桌面)
# 3. 本地视频文件：type="local"，url="视频文件路径" (可以是相对路径——相对这个mock_task.toml文件, 也支持输出到环境变量${HILENS_DATA_DIR}所在目录或子目录)
# 4. http服务：type="webhook", url="http://xxx.xxx" (指的是任务产生的数据上报给某个http服务，此处需填写上传的http服务地址)
[output]
type = "local"
url = "../hilens_data_dir/video_out.mp4"
之后就可以进入项目目录对应用进行构建与运行了：
PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\oxford
PS D:\modelbox-win10-x64-1.5.3\workspace\oxford> .\build_project.sh
PS D:\modelbox-win10-x64-1.5.3\workspace\oxford> .\bin\main.bat
本应用需要事先安装好pillow，可以在workspace\oxford\dependence\modelbox_requirements.txt中填写依赖包，应用运行前会自动检查安装。运行结束后在hilens_data_dir目录下生成了video_out.mp4文件与性能统计文件夹mb_profile：
[2023-06-10 23:38:25,798][ WARN][ iva_config.cc:143 ] update vas url failed. Fault, no vas projectid or iva endpoint
open log file D:/modelbox-win10-x64-1.5.3/workspace/oxford/bin/../hilens_data_dir/log/modelbox.log failed, No error
input dims is:1,224,224,3,
output dims is:1,224,224,3,
output dims is:1,1,
output dims is:1,1,
output dims is:1,1,
output dims is:1,1,
output dims is:1,1,
[h264_mf @ 00000000531920c0] MFT name: 'H264 Encoder MFT'
[2023-06-10 23:40:28,877][ WARN][ffmpeg_video_muxer.cc:78 ] Success: video stream has been written to D:/modelbox-win10-x64-1.5.3/workspace/oxford/hilens_data_dir/video_out.mp4
[2023-06-10 23:40:29,122][ERROR][flow_scheduler.cc:438 ] the scheduler caught an error : Stop operation
Press any key to continue . . .
查看生成的视频以及性能统计文件：
ModelBox是静态图推理，fps取决于耗时最久的功能单元，逐项排查后发现是绘图功能单元，平均耗时84ms，fps = 1000/84 约等于12帧，相比于onnxruntime提升了3~4倍：
可以更换自己的视频进行测试：
小结：我们AI应用的性能瓶颈其实后处理。
【版权声明】本文为华为云社区用户原创内容，转载时必须标注文章的来源（华为云社区）、文章链接、文章作者等基本信息， 否则作者和本社区有权追究责任。如果您发现本社区中有涉嫌抄袭的内容，欢迎发送邮件进行举报，并提供相关证据，一经查实，本社区将立刻删除涉嫌侵权内容，举报邮箱：
cloudbbs@huaweicloud.com
- 点赞
- 收藏
- 关注作者
评论（0）