【ModelArts Course5】Deploying a Trained Model as an Edge Service

举报
杜甫盖房子 发表于 2023/03/14 14:53:32 2023/03/14
【摘要】 Deploying a Trained Model as an Edge Service IntroductionAfter an AI model is developed, you can use it to create an AI application and quickly deploy the application as a real-time service on Mod...

Deploying a Trained Model as an Edge Service

Introduction

After an AI model is developed, you can use it to create an AI application and quickly deploy the application as a real-time service on ModelArts. The API provided by ModelArts real-time service is a standard RESTful API, which can be integrated into production environments conveniently.

You can also download the trained model and inference SDK and deploy it locally as an HTTP service. 【ModelArts Course4】Using Built-in Algorithm for Food Detection describes how to train a model and deploy the trained model as a real-time service, and this document describes how to deploy the trained model as an edge service.

Edge service

The edge service consists of three parts: the model inference SDK on the edge server, the service script on the edge server, and the testing script on the local PC.

The model inference SDK, including trained models and inference code, can be downloaded directly from ModelArts, the service script is built using Flask to set up an HTTP service on the edge server, and the testing script on the local PC loads test images and posts them to the edge server to obtain inference results.

Download SDK

When the training job is completed, click Output Path to go to the OBS directory of the training job output:

You will see a directory organized as this, the model directory is the inference SDK. We can download the SDK from OBS, click Download to download entire folder:

After you download it, you will see a directory organized as below:

├─model
│  ├─models
│      ├─…
│      └─yolo.cpython-36m-x86_64-linux-gnu.so
│  ├─utils_y
│      ├─…
│      └─torch_utils.cpython-36m-x86_64-linux-gnu.so
│  ├─…
│  └─customize_service.py

The customize_service.py is model inference script, and *.so are the files customize_service.py depends on. Then, upload the folder to your edge server.

Environment Preparation

When deploying AI application, some dependences are needed. On the cloud, ModelArts will prepare the runtime environment, including some runtime dependencies and an AI engine with Pytorch and python3.6.

On the edge server, the runtime environment also needs to be prepared. Environments can be created and managed by conda

conda create -n py36 python=3.6.10
conda activate py36
pip install torch==1.7.0 numpy==1.19.0 torchvision==0.8.1 matplotlib==3.3.1 PyYAML==5.3.1

In addition, the dependencies of AI application also need to be prepared:

pip install pandas==1.1.5 requests==2.27.1 pillow==8.4.0 opencv-python==4.7.0.68 tqdm==4.64.1 seaborn==0.11.2 flask==2.0.3

Deploy Service Code

Create a python script in the model directory named application.py:

The application script uses flask to build HTTP services:

from flask import Flask, jsonify, request
from customize_service import YOLOv5

app = Flask(__name__)

yolov5 = YOLOv5(model_name="dishes_det", model_path="best.pt")

@app.route('/dishes_det', methods=['POST'])
def predict():
    file = request.files['file']
    input_dict = {"data": {"file": file}}
    input_data = yolov5._preprocess(input_dict)
    pred = yolov5._inference(input_data)
    out = yolov5._postprocess(pred)
    return jsonify(out)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5005)

Deploy Local Test Code

The application can be accessed using HTTPS on other devices. The inference result is returned in JSON format:

Field Type Description
detection_classes string array List of detected objects, for example, [“dishes”,“cat”]
detection_boxes float array Coordinates of the bounding box, in the format of [ y m i n , x m i n , y m a x , x m a x ] [y_{min},x_{min},y_{max},x_{max}]
detection_scores float array Confidence scores of detected objects, which are used to measure the detection accuracy

On the local device, inference results can be obtained through simple script:

import requests
import json
import cv2

if __name__ == "__main__":
    ip = "xx.xx.xx.xx"  # IP address of the edge server
    port = 5005  # Port set by the service 
    route = "dishes_det"  # Route set by the service
    url = f'http://{ip}:{port}/{route}'
    files = {'file': open('./demo.png', 'rb')}
    response = requests.post(url, files=files)
    img = cv2.imread("./demo.png")
    bbox_data = json.loads(response.content)
    print(bbox_data)
    detection_boxes = bbox_data.get('detection_boxes')
    for box in detection_boxes:
        cv2.rectangle(img, (int(box[1]), int(box[0])), (int(box[3]), int(box[2])), (255, 0, 0), 1)
    cv2.imwrite("./res.jpg", img)

The script output is as follows:

{'detection_boxes': [[99.0, 133.0, 445.0, 492.0], [504.0, 291.0, 741.0, 548.0], [520.0, 90.0, 726.0, 291.0]], 'detection_classes': ['dish', 'dish', 'dish'], 'detection_scores': [0.8971166610717773, 0.8653378486633301, 0.8631448745727539]}

You can also visualize the inference results:

【版权声明】本文为华为云社区用户原创内容,转载时必须标注文章的来源(华为云社区)、文章链接、文章作者等基本信息, 否则作者和本社区有权追究责任。如果您发现本社区中有涉嫌抄袭的内容,欢迎发送邮件进行举报,并提供相关证据,一经查实,本社区将立刻删除涉嫌侵权内容,举报邮箱: cloudbbs@huaweicloud.com
  • 点赞
  • 收藏
  • 关注作者

评论(0

0/1000
抱歉,系统识别当前为高风险访问,暂不支持该操作

全部回复

上滑加载中

设置昵称

在此一键设置昵称,即可参与社区互动!

*长度不超过10个汉字或20个英文字符,设置后3个月内不可修改。

*长度不超过10个汉字或20个英文字符,设置后3个月内不可修改。