unimrcp对接华为云ASR(Linux版)

举报
ASR-beginer 发表于 2024/03/12 09:29:19 2024/03/12
【摘要】 本篇文章提供了unimrcp对接华为云ASR的保姆级教程,根据第一到四章,可从头逐步编译+集成基于华为云ASR的unimrcp系统。同时,本文第五章(太长不看系列)提供了作者修改好的源码,直接一键编译即可。 一、安装unimrcp提前安装必要的依赖项(系统级)autoconf 2.59 or newerautomakelibtool 1.4 or newergccpkg-config下载un...

本篇文章提供了unimrcp对接华为云ASR的保姆级教程,根据第一到四章,可从头逐步编译+集成基于华为云ASR的unimrcp系统(授人以渔)。同时,本文第五章(直接给条鱼)提供了作者修改好的源码,直接一键编译即可。

一、安装unimrcp

  1. 提前安装必要的依赖项(系统级)
autoconf 2.59 or newer
automake
libtool 1.4 or newer
gcc
pkg-config
  1. 下载unimrcp
git clone https://github.com/unispeech/unimrcp.git
  1. 下载unimrcp的依赖库,并安装

进入https://www.unimrcp.org/downloads/dependencies,直接选择最新版本下载

image.png

解压后,进入unimrcp-deps-1.6.0,执行安装命令:

./build-dep-libs.sh
  1. 安装unimrcp
./bootstrap
./configure
make
make install

之后可以在/usr/local/unimrcp 路径下,看到以下内容:

image.png

  1. ASR测试

先启动服务端./unimrcpserver

(base) [root@asr-build bin]# ./unimrcpserver 
2024-03-09 10:23:39:729352 [NOTICE] UniMRCP Server [1.8.0]
2024-03-09 10:23:39:729427 [INFO]   APR [1.5.2]
2024-03-09 10:23:39:729436 [NOTICE] Create MRCP Server
2024-03-09 10:23:39:729459 [NOTICE] Open Config File [/usr/local/unimrcp/conf/unimrcpserver.xml]
2024-03-09 10:23:39:729874 [INFO]   Set Property ip:127.0.0.1
2024-03-09 10:23:39:729888 [INFO]   Register Codec [PCMU]
2024-03-09 10:23:39:729890 [INFO]   Register Codec [PCMA]
2024-03-09 10:23:39:729893 [INFO]   Register Codec [L16]
2024-03-09 10:23:39:729895 [INFO]   Register Codec [G722]
2024-03-09 10:23:39:729911 [NOTICE] Register Resource [speechsynth]
2024-03-09 10:23:39:729916 [NOTICE] Register Resource [speechrecog]
2024-03-09 10:23:39:729918 [NOTICE] Register Resource [recorder]
2024-03-09 10:23:39:729922 [NOTICE] Register Resource [speakverify]
2024-03-09 10:23:39:729925 [INFO]   Register Resource Factory
2024-03-09 10:23:39:729938 [NOTICE] Create SofiaSIP Agent [SIP-Agent-1] [1.12.11-239-g54ef3e2] sip:127.0.0.1:8060;transport=udp,tcp
2024-03-09 10:23:39:729943 [INFO]   Register Signaling Agent [SIP-Agent-1]
2024-03-09 10:23:39:729954 [NOTICE] Create RTSP Server [RTSP-Agent-1] 127.0.0.1:1554 [100] connection timeout [600 sec]
2024-03-09 10:23:39:730021 [INFO]   Register Signaling Agent [RTSP-Agent-1]
2024-03-09 10:23:39:730027 [NOTICE] Create MRCPv2 Agent [MRCPv2-Agent-1] 127.0.0.1:1544 [100]
2024-03-09 10:23:39:730047 [INFO]   Register Connection Agent [MRCPv2-Agent-1]
2024-03-09 10:23:39:730051 [NOTICE] Create Media Engine [Media-Engine-1]
2024-03-09 10:23:39:730058 [INFO]   Register Media Engine [Media-Engine-1]
2024-03-09 10:23:39:730062 [NOTICE] Create RTP Termination Factory 127.0.0.1:[5000,6000]
2024-03-09 10:23:39:730064 [INFO]   Register RTP Termination Factory [RTP-Factory-1]
2024-03-09 10:23:39:730069 [INFO]   Load Plugin [Demo-Synth-1] [/usr/local/unimrcp/plugin/demosynth.so]
2024-03-09 10:23:39:730166 [INFO]   Register MRCP Engine [Demo-Synth-1]
2024-03-09 10:23:39:730171 [INFO]   Load Plugin [Huawei-Recog-1] [/usr/local/unimrcp/plugin/huaweirecog.so]
2024-03-09 10:23:39:730223 [INFO]   Register MRCP Engine [Huawei-Recog-1]
2024-03-09 10:23:39:730226 [INFO]   Load Plugin [Demo-Verifier-1] [/usr/local/unimrcp/plugin/demoverifier.so]
2024-03-09 10:23:39:730276 [INFO]   Register MRCP Engine [Demo-Verifier-1]
2024-03-09 10:23:39:730279 [INFO]   Load Plugin [Recorder-1] [/usr/local/unimrcp/plugin/mrcprecorder.so]
2024-03-09 10:23:39:730323 [INFO]   Register MRCP Engine [Recorder-1]
2024-03-09 10:23:39:730333 [INFO]   Register RTP Settings [RTP-Settings-1]
2024-03-09 10:23:39:730338 [NOTICE] Create MRCPv2 Profile [uni2]
2024-03-09 10:23:39:730344 [INFO]   Associate Resource [speechsynth] to Engine [Demo-Synth-1] in Profile [uni2]
2024-03-09 10:23:39:730347 [INFO]   Associate Resource [speechrecog] to Engine [Huawei-Recog-1] in Profile [uni2]
2024-03-09 10:23:39:730349 [INFO]   Associate Resource [recorder] to Engine [Recorder-1] in Profile [uni2]
2024-03-09 10:23:39:730351 [INFO]   Associate Resource [speakverify] to Engine [Demo-Verifier-1] in Profile [uni2]
2024-03-09 10:23:39:730354 [INFO]   Register Profile [uni2]
2024-03-09 10:23:39:730356 [NOTICE] Create MRCPv1 Profile [uni1]
2024-03-09 10:23:39:730358 [INFO]   Associate Resource [speechsynth] to Engine [Demo-Synth-1] in Profile [uni1]
2024-03-09 10:23:39:730360 [INFO]   Associate Resource [speechrecog] to Engine [Huawei-Recog-1] in Profile [uni1]
2024-03-09 10:23:39:730362 [INFO]   Associate Resource [recorder] to Engine [Recorder-1] in Profile [uni1]
2024-03-09 10:23:39:730364 [INFO]   Associate Resource [speakverify] to Engine [Demo-Verifier-1] in Profile [uni1]
2024-03-09 10:23:39:730367 [INFO]   Register Profile [uni1]
2024-03-09 10:23:39:730370 [INFO]   Start Task [MRCP Server]
>2024-03-09 10:23:39:730456 [INFO]   Open Engine [Demo-Verifier-1]
2024-03-09 10:23:39:730471 [INFO]   Start Task [Demo Verifier Engine]
2024-03-09 10:23:39:730526 [INFO]   Open Engine [Huawei-Recog-1]
2024-03-09 10:23:39:730530 [INFO]   Start Task [Huawei Recog Engine]
2024-03-09 10:23:39:730552 [INFO]   Open Engine [Recorder-1]
2024-03-09 10:23:39:730555 [INFO]   Open Engine [Demo-Synth-1]
2024-03-09 10:23:39:730557 [INFO]   Start Task [Demo Synth Engine]
2024-03-09 10:23:39:730580 [INFO]   Start Task [SIP-Agent-1]
2024-03-09 10:23:39:730599 [INFO]   Start Task [RTSP-Agent-1]
2024-03-09 10:23:39:730619 [INFO]   Start Task [MRCPv2-Agent-1]
2024-03-09 10:23:39:730642 [INFO]   Start Task [Media-Engine-1]
sres: /etc/resolv.conf: unknown option                      
2024-03-09 10:23:39:731672 [NOTICE] MRCP Server Started

再启动客户端,测试ASR

./umc
run recog
>run recog
>[1]
2024-03-09 10:24:46:654383 [NOTICE] Create MRCP Handle 0x7f902c035630 [uni2]
2024-03-09 10:24:46:654420 [INFO]   Create Channel umc-1 <new>
2024-03-09 10:24:46:654466 [INFO]   Receive App Request umc-1 <new> [2]
2024-03-09 10:24:46:654535 [INFO]   Add MRCP Handle umc-1 <new>
2024-03-09 10:24:46:654573 [NOTICE] Add Control Channel umc-1 <new@speechrecog>
2024-03-09 10:24:46:656847 [INFO]   Send Offer umc-1 <new> [c:1 a:1 v:0] to 127.0.0.1:8060
2024-03-09 10:24:46:656879 [INFO]   Local SDP umc-1 <new>
v=0
o=UniMRCPClient 0 0 IN IP4 127.0.0.1
s=-
c=IN IP4 127.0.0.1
t=0 0
m=application 9 TCP/MRCPv2 1
a=setup:active
a=connection:new
a=resource:speechrecog
a=cmid:1
m=audio 4000 RTP/AVP 0 8 9 96 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:9 G722/8000
a=rtpmap:96 L16/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendonly
a=ptime:20
a=mid:1

2024-03-09 10:24:46:657606 [INFO]   Receive SIP Event [nua_i_state] Status 0 INVITE sent [SIP-Agent-1]
2024-03-09 10:24:46:657624 [NOTICE] SIP Call State umc-1 [calling]
2024-03-09 10:24:46:661600 [INFO]   Receive SIP Event [nua_r_invite] Status 200 OK [SIP-Agent-1]
2024-03-09 10:24:46:661658 [INFO]   Receive SIP Event [nua_i_state] Status 200 OK [SIP-Agent-1]
2024-03-09 10:24:46:661664 [NOTICE] SIP Call State umc-1 [ready]
2024-03-09 10:24:46:661668 [INFO]   Remote SDP umc-1 <new>
v=0
o=UniMRCPServer 8207428304472510678 6899681545498287106 IN IP4 192.168.34.32
s=-
c=IN IP4 127.0.0.1
t=0 0
m=application 1544 TCP/MRCPv2 1
a=setup:passive
a=connection:new
a=channel:328a56e4ddbc11ee@speechrecog
a=cmid:1
m=audio 5000 RTP/AVP 0 101
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=recvonly
a=ptime:20
a=mid:1

2024-03-09 10:24:46:661738 [INFO]   Receive SIP Event [nua_i_active] Status 200 Call active [SIP-Agent-1]
2024-03-09 10:24:46:661756 [INFO]   Receive Answer umc-1 <new> [c:1 a:1 v:0] Status 200
2024-03-09 10:24:46:661892 [NOTICE] Established TCP/MRCPv2 Connection 127.0.0.1:59830 <-> 127.0.0.1:1544
2024-03-09 10:24:46:661906 [INFO]   Add Control Channel <328a56e4ddbc11ee@speechrecog> 127.0.0.1:59830 <-> 127.0.0.1:1544 [1]
2024-03-09 10:24:46:666844 [INFO]   Enable RTP Session 127.0.0.1:4000
2024-03-09 10:24:46:666873 [INFO]   Open RTP Transmitter 127.0.0.1:4000 -> 127.0.0.1:5000
2024-03-09 10:24:46:666878 [INFO]   Media Path umc-1 Source->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMU/8000/1]->Sink
2024-03-09 10:24:46:666900 [INFO]   Raise App Response umc-1 <328a56e4ddbc11ee> [2] SUCCESS [0]
2024-03-09 10:24:46:666975 [INFO]   Receive App MRCP Request umc-1 <328a56e4ddbc11ee>
2024-03-09 10:24:46:666979 [INFO]   Send MRCP Request umc-1 <328a56e4ddbc11ee@speechrecog> [1]
2024-03-09 10:24:46:667028 [INFO]   Send MRCPv2 Data 127.0.0.1:59830 <-> 127.0.0.1:1544 [442 bytes]
MRCP/2.0 442 DEFINE-GRAMMAR 1
Channel-Identifier: 328a56e4ddbc11ee@speechrecog
Content-Type: application/srgs+xml
Content-Id: request1@form-level
Content-Length: 269

<?xml version="1.0"?>
<grammar xmlns="http://www.w3.org/2001/06/grammar" xml:lang="en-US" version="1.0" mode="voice" root="digit">
  <rule id="digit">
    <one-of>
      <item>one</item>
      <item>two</item>
      <item>three</item>
    </one-of>
  </rule>
</grammar>
2024-03-09 10:24:46:667298 [INFO]   Receive MRCPv2 Data 127.0.0.1:59830 <-> 127.0.0.1:1544 [112 bytes]
MRCP/2.0 112 1 200 COMPLETE
Channel-Identifier: 328a56e4ddbc11ee@speechrecog
Completion-Cause: 000 success


2024-03-09 10:24:46:667336 [INFO]   Raise App MRCP Response umc-1 <328a56e4ddbc11ee>
2024-03-09 10:24:46:667403 [INFO]   Receive App MRCP Request umc-1 <328a56e4ddbc11ee>
2024-03-09 10:24:46:667406 [INFO]   Send MRCP Request umc-1 <328a56e4ddbc11ee@speechrecog> [2]
2024-03-09 10:24:46:667414 [INFO]   Set [/usr/local/unimrcp/data/one-8kHz.pcm] as Speech Source
2024-03-09 10:24:46:667428 [INFO]   Send MRCPv2 Data 127.0.0.1:59830 <-> 127.0.0.1:1544 [304 bytes]
MRCP/2.0 304 RECOGNIZE 2
Channel-Identifier: 328a56e4ddbc11ee@speechrecog
Content-Type: text/uri-list
Cancel-If-Queue: false
No-Input-Timeout: 5000
Recognition-Timeout: 10000
Start-Input-Timers: true
Confidence-Threshold: 0.5
Save-Waveform: true
Content-Length: 27

session:request1@form-level
2024-03-09 10:24:46:667664 [INFO]   Receive MRCPv2 Data 127.0.0.1:59830 <-> 127.0.0.1:1544 [83 bytes]
MRCP/2.0 83 2 200 IN-PROGRESS
Channel-Identifier: 328a56e4ddbc11ee@speechrecog


2024-03-09 10:24:46:667686 [INFO]   Raise App MRCP Response umc-1 <328a56e4ddbc11ee>
2024-03-09 10:24:47:541082 [INFO]   Receive MRCPv2 Data 127.0.0.1:59830 <-> 127.0.0.1:1544 [94 bytes]
MRCP/2.0 94 START-OF-INPUT 2 IN-PROGRESS
Channel-Identifier: 328a56e4ddbc11ee@speechrecog


2024-03-09 10:24:47:541165 [INFO]   Raise App MRCP Event umc-1 <328a56e4ddbc11ee>
2024-03-09 10:24:48:011172 [INFO]   Receive MRCPv2 Data 127.0.0.1:59830 <-> 127.0.0.1:1544 [393 bytes]
MRCP/2.0 393 RECOGNITION-COMPLETE 2 COMPLETE
Channel-Identifier: 328a56e4ddbc11ee@speechrecog
Completion-Cause: 000 success
Content-Type: application/x-nlsml
Content-Length: 208

<?xml version="1.0"?>
<result> 
  <interpretation grammar="session:request1@form-level.store" confidence="0.97">
    <instance>one</instance>
    <input mode="speech">one</input>
  </interpretation>
</result>
2024-03-09 10:24:48:011247 [INFO]   Raise App MRCP Event umc-1 <328a56e4ddbc11ee>
2024-03-09 10:24:48:011354 [INFO]   Interpretation[0].confidence: 0.97
2024-03-09 10:24:48:011363 [INFO]   Interpretation[0].grammar: session:request1@form-level.store
2024-03-09 10:24:48:011380 [INFO]   Interpretation[0].instance[0]: one
2024-03-09 10:24:48:011386 [INFO]   Interpretation[0].input: one
2024-03-09 10:24:48:011388 [INFO]   Interpretation[0].input.mode: speech
2024-03-09 10:24:48:011391 [INFO]   Interpretation[0].input.confidence: 1.00
2024-03-09 10:24:48:011412 [INFO]   Receive App Request umc-1 <328a56e4ddbc11ee> [1]
2024-03-09 10:24:48:011424 [INFO]   Terminate Session umc-1 <328a56e4ddbc11ee>
2024-03-09 10:24:48:011454 [INFO]   Remove Control Channel <328a56e4ddbc11ee@speechrecog> [0]
2024-03-09 10:24:48:011459 [INFO]   Close TCP/MRCPv2 Connection 127.0.0.1:59830 <-> 127.0.0.1:1544
2024-03-09 10:24:48:011791 [INFO]   Receive SIP Event [nua_r_bye] Status 200 OK [SIP-Agent-1]
2024-03-09 10:24:48:011805 [INFO]   Receive SIP Event [nua_i_state] Status 200 to BYE [SIP-Agent-1]
2024-03-09 10:24:48:011810 [NOTICE] SIP Call State umc-1 [terminated]
2024-03-09 10:24:48:011871 [INFO]   Session Terminated umc-1 <328a56e4ddbc11ee>
2024-03-09 10:24:48:016774 [INFO]   Close RTP Transmitter 127.0.0.1:4000 -> 127.0.0.1:5000 [s:67 o:10720]
2024-03-09 10:24:48:016791 [INFO]   Remove RTP Session 127.0.0.1:4000
2024-03-09 10:24:48:016831 [NOTICE] Destroy TCP/MRCPv2 Connection 127.0.0.1:59830 <-> 127.0.0.1:1544
2024-03-09 10:24:48:016851 [INFO]   Remove MRCP Handle umc-1 <328a56e4ddbc11ee>
2024-03-09 10:24:48:016855 [INFO]   Raise App Response umc-1 <328a56e4ddbc11ee> [1] SUCCESS [0]
2024-03-09 10:24:48:016953 [NOTICE] Destroy MRCP Handle umc-1

第124~130行即为模拟的识别结果

image-20240309102906236

二、安装华为ASR SDK

华为ASR C++(Linux版)SDK安装过程参考之前的文章:https://bbs.huaweicloud.com/blogs/392949

这里推荐一键安装,因为本SDK将所有依赖均打包在仓库中,可以在任意Linux机器上编译通过(g++ 4.8.5及以上,cmake版本至少是3.14)

三、unimrcp集成华为ASR SDK

本章节用从底到高的顺序进行记录,即先修改底层代码/配置,再修改顶层代码

  1. 将(上个步骤 安装华为ASR SDK)编译好的SDK打包起来,并命名为huaweicloud_asr,作为备用,其代码结构如下:

image.png

  1. 进入unimrcp/plugins,调整代码结构,如下图所示
cd umimrcp/plugins
mv demo-recog huawei-recog    # 将原始的demo-recog重命名成huawei-recog
mv huawei-recog/src/demo_recog_engine.c huawei-recog/src/huawei_recog_engine.cc
cp -r path_to_sdk/huaweicloud_asr huawei-recog/huaweicloud_asr   # 将华为asr sdk拷贝到umimrcp/plugins/huawei-recog/huaweicloud_asr处


image.png

  1. 修改huawei-recog/src/huawei_recog_engine.cc文件,具体内容如下
/*
 * Copyright 2008-2015 Arsen Chaloyan
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

/*
 * Mandatory rules concerning plugin implementation.
 * 1. Each plugin MUST implement a plugin/engine creator function
 *    with the exact signature and name (the main entry point)
 *        MRCP_PLUGIN_DECLARE(mrcp_engine_t*) mrcp_plugin_create(apr_pool_t *pool)
 * 2. Each plugin MUST declare its version number
 *        MRCP_PLUGIN_VERSION_DECLARE
 * 3. One and only one response MUST be sent back to the received request.
 * 4. Methods (callbacks) of the MRCP engine channel MUST not block.
 *   (asynchronous response can be sent from the context of other thread)
 * 5. Methods (callbacks) of the MPF engine stream MUST not block.
 */

extern "C" {
#include "mrcp_recog_engine.h"
#include "mpf_activity_detector.h"
#include "apt_consumer_task.h"
#include "apt_log.h"
}  // extern C

#include "Utils.h"
#include "RasrListener.h"
#include "RasrClient.h"
#include "gflags/gflags.h"
#define RECOG_ENGINE_TASK_NAME "Huawei Recog Engine"
typedef struct huawei_recog_engine_t huawei_recog_engine_t;
typedef struct huawei_recog_channel_t huawei_recog_channel_t;
typedef struct huawei_recog_msg_t huawei_recog_msg_t;


DEFINE_string(ak, "", "access key");
DEFINE_string(sk, "", "secrect key");
// region, for example cn-east-3, cn-north-4
DEFINE_string(region, "cn-east-3", "project region, such as cn-east-3");
// projectId, refer to https://support.huaweicloud.com/api-sis/sis_03_0008.html
DEFINE_string(projectId, "", "project id");
// endpoint, relevant to region, sis-ext.${region}.myhuaweicloud.com
DEFINE_string(endpoint, "", "service endpoint");
DEFINE_string(audioFormat, "pcm8k16bit", "such pcm16k16bit alaw16k16bit etc.");
DEFINE_string(property, "chinese_8k_general", "");
DEFINE_string(asr_mode, "continue", "continue, short , sentence");
DEFINE_int32(readTimeOut, 20000, "read time out, default 20s");
DEFINE_int32(connectTimeOut, 20000, "connecting time out, default 20s");
DEFINE_int32(vadHead, 10000, "read time out, default 20s");
DEFINE_int32(vadTail, 800, "connecting time out, default 20s");
DEFINE_int32(maxSeconds, 60, "connecting time out, default 20s");
/** Declare this macro to set plugin version */
MRCP_PLUGIN_VERSION_DECLARE

/**
 * Declare this macro to use log routine of the server, plugin is loaded from.
 * Enable/add the corresponding entry in logger.xml to set a cutsom log source priority.
 *    <source name="RECOG-PLUGIN" priority="DEBUG" masking="NONE"/>
 */
MRCP_PLUGIN_LOG_SOURCE_IMPLEMENT(RECOG_PLUGIN, "RECOG-PLUGIN")

/** Use custom log source mark */
#define RECOG_LOG_MARK APT_LOG_MARK_DECLARE(RECOG_PLUGIN)

enum class Status :int {
    initial = 0,
    opened = 1,
    started = 2,
    voice_start = 3,
    recognizing = 4,
    voice_end = 5,
    exceeded_silence = 6,
    ended = 7,
    closed = 8
};

class CallBack : public RasrListener {
public:
    CallBack() {
        result.clear();
        status = Status::initial;
    }
    void OnOpen()
    {   
        status = Status::opened;
        apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "now rasr Connect success");
    }
    void OnStart(std::string text)
    {
        status = Status::started;
        apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "now rasr receive start response: %s", text.c_str());
    }
    void OnResp(std::string text)
    {
        apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "user [%s] rasr receive %s", userMeta_.c_str(), text.c_str());
        result.assign(text);
    }
    void OnEnd(std::string text)
    {
        status = Status::ended;
        apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "now rasr receive end response: %s", text.c_str());
    }
    void OnClose()
    {
        status = Status::closed;
        apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "now rasr receive Close");
    }
    void OnError(std::string text)
    {
        apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "now rasr receive error: %s", text.c_str());
    }
    void OnEvent(std::string text)
    {
        // EXCEED_SILENCE
        if (text.find("VOICE_START") != std::string::npos) {
            status = Status::voice_start;
        } else if (text.find("EXCEEDED_SILENCE") != std::string::npos) {
            status = Status::exceeded_silence;
        } else if (text.find("VOICE_END") != std::string::npos) {
            status = Status::voice_end;
        }
        apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "now rasr receive event: %s", text.c_str());
    }
    void SetUserInfo(const std::string &str)
    {
        userMeta_ = str;
    }
    const std::string Result() {
        return result;
    }
    Status GetStatus() {
        return status;
    }
    void SetStatus(Status in) {
        status = in;
    }
private:
    std::string userMeta_;
    std::string result;
    Status status;
};

/** Declaration of recognizer engine methods */
static apt_bool_t huawei_recog_engine_destroy(mrcp_engine_t *engine);
static apt_bool_t huawei_recog_engine_open(mrcp_engine_t *engine);
static apt_bool_t huawei_recog_engine_close(mrcp_engine_t *engine);
static mrcp_engine_channel_t *huawei_recog_engine_channel_create(mrcp_engine_t *engine, apr_pool_t *pool);

static const struct mrcp_engine_method_vtable_t engine_vtable = {huawei_recog_engine_destroy,
    huawei_recog_engine_open,
    huawei_recog_engine_close,
    huawei_recog_engine_channel_create};

/** Declaration of recognizer channel methods */
static apt_bool_t huawei_recog_channel_destroy(mrcp_engine_channel_t *channel);
static apt_bool_t huawei_recog_channel_open(mrcp_engine_channel_t *channel);
static apt_bool_t huawei_recog_channel_close(mrcp_engine_channel_t *channel);
static apt_bool_t huawei_recog_channel_request_process(mrcp_engine_channel_t *channel, mrcp_message_t *request);

static const struct mrcp_engine_channel_method_vtable_t channel_vtable = {huawei_recog_channel_destroy,
    huawei_recog_channel_open,
    huawei_recog_channel_close,
    huawei_recog_channel_request_process};

/** Declaration of recognizer audio stream methods */
static apt_bool_t huawei_recog_stream_destroy(mpf_audio_stream_t *stream);
static apt_bool_t huawei_recog_stream_open(mpf_audio_stream_t *stream, mpf_codec_t *codec);
static apt_bool_t huawei_recog_stream_close(mpf_audio_stream_t *stream);
static apt_bool_t huawei_recog_stream_write(mpf_audio_stream_t *stream, const mpf_frame_t *frame);

static const mpf_audio_stream_vtable_t audio_stream_vtable = {huawei_recog_stream_destroy,
    NULL,
    NULL,
    NULL,
    huawei_recog_stream_open,
    huawei_recog_stream_close,
    huawei_recog_stream_write,
    NULL};

/** Declaration of huawei recognizer engine */
struct huawei_recog_engine_t {
    apt_consumer_task_t *task;
};

/** Declaration of huawei recognizer channel */
struct huawei_recog_channel_t {
    /** Back pointer to engine */
    huawei_recog_engine_t *huawei_engine;
    /** Engine channel base */
    mrcp_engine_channel_t *channel;

    /** Active (in-progress) recognition request */
    mrcp_message_t *recog_request;
    /** Pending stop response */
    mrcp_message_t *stop_response;
    /** Indicates whether input timers are started */
    apt_bool_t timers_started;
    /** Voice activity detector */
    mpf_activity_detector_t *detector;
    /** File to write utterance to */
    FILE *audio_out;

    /** huawei rasr client */
    speech::huawei_asr::RasrClient* rasr_client;  // 华为SDK的属性
    CallBack* call_back;
    ~huawei_recog_channel_t() {
        if (rasr_client) {
            delete rasr_client;
        }
        if (call_back) {
            delete call_back;
        }
    }
};

typedef enum {
    HUAWEI_RECOG_MSG_OPEN_CHANNEL,
    HUAWEI_RECOG_MSG_CLOSE_CHANNEL,
    HUAWEI_RECOG_MSG_REQUEST_PROCESS
} huawei_recog_msg_type_e;

/** Declaration of demo recognizer task message */
struct huawei_recog_msg_t {
    huawei_recog_msg_type_e type;
    mrcp_engine_channel_t *channel;
    mrcp_message_t *request;
};

static apt_bool_t huawei_recog_msg_signal(
    huawei_recog_msg_type_e type, mrcp_engine_channel_t *channel, mrcp_message_t *request);
static apt_bool_t huawei_recog_msg_process(apt_task_t *task, apt_task_msg_t *msg);

/** Create huawei recognizer engine */
MRCP_PLUGIN_DECLARE(mrcp_engine_t *) mrcp_plugin_create(apr_pool_t *pool)
{
    huawei_recog_engine_t *huawei_engine = (huawei_recog_engine_t *)apr_palloc(pool, sizeof(huawei_recog_engine_t));
    apt_task_t *task;
    apt_task_vtable_t *vtable;
    apt_task_msg_pool_t *msg_pool;

    msg_pool = apt_task_msg_pool_create_dynamic(sizeof(huawei_recog_msg_t), pool);
    huawei_engine->task = apt_consumer_task_create(huawei_engine, msg_pool, pool);
    if (!huawei_engine->task) {
        return NULL;
    }
    task = apt_consumer_task_base_get(huawei_engine->task);
    apt_task_name_set(task, RECOG_ENGINE_TASK_NAME);
    vtable = apt_task_vtable_get(task);
    if (vtable) {
        vtable->process_msg = huawei_recog_msg_process;
    }

    /* create engine base */
    return mrcp_engine_create(MRCP_RECOGNIZER_RESOURCE, /* MRCP resource identifier */
        huawei_engine,                                  /* object to associate */
        &engine_vtable,                                 /* virtual methods table of engine */
        pool);                                          /* pool to allocate memory from */
}

/** Destroy recognizer engine */
static apt_bool_t huawei_recog_engine_destroy(mrcp_engine_t *engine)
{
    huawei_recog_engine_t *huawei_engine = (huawei_recog_engine_t *)engine->obj;
    if (huawei_engine->task) {
        apt_task_t *task = apt_consumer_task_base_get(huawei_engine->task);
        apt_task_destroy(task);
        huawei_engine->task = NULL;
    }
    return TRUE;
}

/** Open recognizer engine */
static apt_bool_t huawei_recog_engine_open(mrcp_engine_t *engine)
{
    huawei_recog_engine_t *huawei_engine = (huawei_recog_engine_t *)engine->obj;
    if (huawei_engine->task) {
        apt_task_t *task = apt_consumer_task_base_get(huawei_engine->task);
        apt_task_start(task);
    }

    return mrcp_engine_open_respond(engine, TRUE);
}

/** Close recognizer engine */
static apt_bool_t huawei_recog_engine_close(mrcp_engine_t *engine)
{
    huawei_recog_engine_t *huawei_engine = (huawei_recog_engine_t *)engine->obj;
    if (huawei_engine->task) {
        apt_task_t *task = apt_consumer_task_base_get(huawei_engine->task);
        apt_task_terminate(task, TRUE);
    }

    return mrcp_engine_close_respond(engine);
}

static mrcp_engine_channel_t *huawei_recog_engine_channel_create(mrcp_engine_t *engine, apr_pool_t *pool)
{
    mpf_stream_capabilities_t *capabilities;
    mpf_termination_t *termination;

    /* create huawei recog channel */
    huawei_recog_channel_t *recog_channel = (huawei_recog_channel_t *)apr_palloc(pool, sizeof(huawei_recog_channel_t));
    recog_channel->huawei_engine = (huawei_recog_engine_t *)engine->obj;
    recog_channel->recog_request = nullptr;
    recog_channel->stop_response = nullptr;
    recog_channel->detector = mpf_activity_detector_create(pool);
    recog_channel->audio_out = nullptr;
    recog_channel->rasr_client = nullptr;
    recog_channel->call_back = nullptr;
    
    capabilities = mpf_sink_stream_capabilities_create(pool);
    mpf_codec_capabilities_add(&capabilities->codecs, MPF_SAMPLE_RATE_8000 | MPF_SAMPLE_RATE_16000, "LPCM");

    /* create media termination */
    termination = mrcp_engine_audio_termination_create(recog_channel, /* object to associate */
        &audio_stream_vtable,                                         /* virtual methods table of audio stream */
        capabilities,                                                 /* stream capabilities */
        pool);                                                        /* pool to allocate memory from */

    /* create engine channel base */
    recog_channel->channel = mrcp_engine_channel_create(engine, /* engine */
        &channel_vtable,                                        /* virtual methods table of engine channel */
        recog_channel,                                          /* object to associate */
        termination,                                            /* associated media termination */
        pool);                                                  /* pool to allocate memory from */
    const apt_dir_layout_t *dir_layout = engine->dir_layout;
    // 获取asr配置文件 以下两行均可以正常获取
    // const char* asr_conf_path = apt_dir_layout_path_compose(dir_layout, APT_LAYOUT_CONF_DIR, "asr_info.conf", pool);
    const char* asr_conf_path = apt_confdir_filepath_get(dir_layout, "asr_info.conf", pool);
    apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "set asr info from %s", asr_conf_path);
    google::SetCommandLineOption("flagfile", asr_conf_path);
    return recog_channel->channel;
}

/** Destroy engine channel */
static apt_bool_t huawei_recog_channel_destroy(mrcp_engine_channel_t *channel)
{
    /* nothing to destrtoy */
    return TRUE;
}

/** Open engine channel (asynchronous response MUST be sent)*/
static apt_bool_t huawei_recog_channel_open(mrcp_engine_channel_t *channel)
{
    if (channel->attribs) {
        /* process attributes */
        const apr_array_header_t *header = apr_table_elts(channel->attribs);
        apr_table_entry_t *entry = (apr_table_entry_t *)header->elts;
        int i;
        for (i = 0; i < header->nelts; i++) {
            apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "Attrib name [%s] value [%s]", entry[i].key, entry[i].val);
        }
    }

    return huawei_recog_msg_signal(HUAWEI_RECOG_MSG_OPEN_CHANNEL, channel, NULL);
}

/** Close engine channel (asynchronous response MUST be sent)*/
static apt_bool_t huawei_recog_channel_close(mrcp_engine_channel_t *channel)
{
    return huawei_recog_msg_signal(HUAWEI_RECOG_MSG_CLOSE_CHANNEL, channel, NULL);
}

/** Process MRCP channel request (asynchronous response MUST be sent)*/
static apt_bool_t huawei_recog_channel_request_process(mrcp_engine_channel_t *channel, mrcp_message_t *request)
{
    return huawei_recog_msg_signal(HUAWEI_RECOG_MSG_REQUEST_PROCESS, channel, request);
}

/** Process RECOGNIZE request */
static apt_bool_t huawei_recog_channel_recognize(
    mrcp_engine_channel_t *channel, mrcp_message_t *request, mrcp_message_t *response)
{
    /* process RECOGNIZE request */
    mrcp_recog_header_t *recog_header;
    huawei_recog_channel_t *recog_channel = (huawei_recog_channel_t *)channel->method_obj;
    const mpf_codec_descriptor_t *descriptor = mrcp_engine_sink_stream_codec_get(channel);

    if (!descriptor) {
        apt_log(RECOG_LOG_MARK,
            APT_PRIO_WARNING,
            "Failed to Get Codec Descriptor " APT_SIDRES_FMT,
            MRCP_MESSAGE_SIDRES(request));
        response->start_line.status_code = MRCP_STATUS_CODE_METHOD_FAILED;
        return FALSE;
    }

    recog_channel->timers_started = TRUE;

    /* get recognizer header */ 
    // 通过请求header设置vad断句参数  unimrcp提供的断句不够精准,推荐用华为云asr自带的断句能力
    recog_header = (mrcp_recog_header_t *)mrcp_resource_header_get(request);
    if (recog_header) {
        if (mrcp_resource_header_property_check(request, RECOGNIZER_HEADER_START_INPUT_TIMERS) == TRUE) {
            recog_channel->timers_started = recog_header->start_input_timers;
        }
        if (mrcp_resource_header_property_check(request, RECOGNIZER_HEADER_NO_INPUT_TIMEOUT) == TRUE) {
            mpf_activity_detector_noinput_timeout_set(recog_channel->detector, recog_header->no_input_timeout);
        }
        if (mrcp_resource_header_property_check(request, RECOGNIZER_HEADER_SPEECH_COMPLETE_TIMEOUT) == TRUE) {
            mpf_activity_detector_silence_timeout_set(recog_channel->detector, recog_header->speech_complete_timeout);
        }
    }

    if (!recog_channel->audio_out) {
        const apt_dir_layout_t *dir_layout = channel->engine->dir_layout;
        char *file_name = apr_psprintf(
            channel->pool, "utter-%dkHz-%s.pcm", descriptor->sampling_rate / 1000, request->channel_id.session_id.buf);
        char *file_path = apt_vardir_filepath_get(dir_layout, file_name, channel->pool);
        if (file_path) {
            apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "Open Utterance Output File [%s] for Writing", file_path);
            recog_channel->audio_out = fopen(file_path, "wb");
            if (!recog_channel->audio_out) {
                apt_log(RECOG_LOG_MARK,
                    APT_PRIO_WARNING,
                    "Failed to Open Utterance Output File [%s] for Writing",
                    file_path);
            }
        }
    }

    // 初始化ASR引擎
    speech::huawei_asr::AuthInfo authInfo(FLAGS_ak, FLAGS_sk, FLAGS_region, FLAGS_projectId, FLAGS_endpoint);
    // config Connect parameter
    speech::huawei_asr::HttpConfig httpConfig;
    httpConfig.SetReadTimeout(FLAGS_readTimeOut);
    httpConfig.SetConnectTimeout(FLAGS_connectTimeOut);

    // config callback, callback function are optional, if not set, it will use function in RasrListener
    speech::huawei_asr::WebsocketService::ptr websocketServicePtr =
        websocketpp::lib::make_shared<speech::huawei_asr::WebsocketService>();
    CallBack* callBack = new CallBack();
    callBack->SetUserInfo("User Info");
    websocketServicePtr->SetCallBack(callBack); // Connect success callback
    // step1 create client
    speech::huawei_asr::RasrClient* rasrClient = new speech::huawei_asr::RasrClient(authInfo, websocketServicePtr, httpConfig); 

    recog_channel->rasr_client = rasrClient;
    recog_channel->call_back = callBack;

    // 判断指针是否创建成功
    if (rasrClient == NULL) {
        printf("create rasr client failed\n");
        return FALSE;
    }

    // step2 connect, just select one mode, the following is continue stream connect.
    if (FLAGS_asr_mode == "continue") {
        rasrClient->ContinueStreamConnect();
    } else if (FLAGS_asr_mode == "short") {
        rasrClient->ShortStreamConnect();
    } else {
        rasrClient->SentenceStreamConnect();
    }

    // step3 construct request params
    speech::huawei_asr::RasrRequest rasr_request(FLAGS_audioFormat, FLAGS_property);
    // set whether to add punctuation, yes or no, default no, optional operation.
    rasr_request.SetPunc("no");
    // set whether to transcribe number into arabic numerals, yes or no, default yes,optional operation.
    rasr_request.SetDigitNorm("yes");
    // set vad head, max silent head, [0, 60000], default 10000, optional operation.
    rasr_request.SetVadHead(FLAGS_vadHead);
    // set vad tail, max silent tail, [0, 3000], default 500, optional operation.
    rasr_request.SetVadTail(FLAGS_vadTail);
    // set max seconds of one sentence, [1, 60], default 30, optional operation.
    rasr_request.SetMaxSeconds(FLAGS_maxSeconds);
    // set whether to return intermediate result, yes or no, default no. optional operation.
    rasr_request.SetIntermediateResult("no");
    // set whether to return word_info, yes or no, default no. optional operation.
    rasr_request.SetNeedWordInfo("no");
    // set vocabulary_id, it should be filled only if it exists or it will report error
    // request.SetVocabularyId("");

    // step4 send start
    rasrClient->SendStart(rasr_request);

    response->start_line.request_state = MRCP_REQUEST_STATE_INPROGRESS;
    /* send asynchronous response */
    mrcp_engine_channel_message_send(channel, response);
    recog_channel->recog_request = request;
    return TRUE;
}

/** Process STOP request */
static apt_bool_t huawei_recog_channel_stop(
    mrcp_engine_channel_t *channel, mrcp_message_t *request, mrcp_message_t *response)
{
    /* process STOP request */
    huawei_recog_channel_t *recog_channel = (huawei_recog_channel_t *)channel->method_obj;
    /* store STOP request, make sure there is no more activity and only then send the response */
    recog_channel->stop_response = response;
    return TRUE;
}

/** Process START-INPUT-TIMERS request */
static apt_bool_t huawei_recog_channel_timers_start(
    mrcp_engine_channel_t *channel, mrcp_message_t *request, mrcp_message_t *response)
{
    huawei_recog_channel_t *recog_channel = (huawei_recog_channel_t *)channel->method_obj;
    recog_channel->timers_started = TRUE;
    return mrcp_engine_channel_message_send(channel, response);
}

/** Dispatch MRCP request */
static apt_bool_t huawei_recog_channel_request_dispatch(mrcp_engine_channel_t *channel, mrcp_message_t *request)
{
    apt_bool_t processed = FALSE;
    mrcp_message_t *response = mrcp_response_create(request, request->pool);
    switch (request->start_line.method_id) {
        case RECOGNIZER_SET_PARAMS:
            break;
        case RECOGNIZER_GET_PARAMS:
            break;
        case RECOGNIZER_DEFINE_GRAMMAR:
            break;
        case RECOGNIZER_RECOGNIZE:
            processed = huawei_recog_channel_recognize(channel, request, response);
            break;
        case RECOGNIZER_GET_RESULT:
            break;
        case RECOGNIZER_START_INPUT_TIMERS:
            processed = huawei_recog_channel_timers_start(channel, request, response);
            break;
        case RECOGNIZER_STOP:
            processed = huawei_recog_channel_stop(channel, request, response);
            break;
        default:
            break;
    }
    if (processed == FALSE) {
        /* send asynchronous response for not handled request */
        mrcp_engine_channel_message_send(channel, response);
    }
    return TRUE;
}

/** Callback is called from MPF engine context to destroy any additional data associated with audio stream */
static apt_bool_t huawei_recog_stream_destroy(mpf_audio_stream_t *stream)
{
    return TRUE;
}

/** Callback is called from MPF engine context to perform any action before open */
static apt_bool_t huawei_recog_stream_open(mpf_audio_stream_t *stream, mpf_codec_t *codec)
{
    return TRUE;
}

/** Callback is called from MPF engine context to perform any action after close */
static apt_bool_t huawei_recog_stream_close(mpf_audio_stream_t *stream)
{
    return TRUE;
}

/* Raise huawei START-OF-INPUT event */
static apt_bool_t huawei_recog_start_of_input(huawei_recog_channel_t *recog_channel)
{
    /* create START-OF-INPUT event */
    mrcp_message_t *message =
        mrcp_event_create(recog_channel->recog_request, RECOGNIZER_START_OF_INPUT, recog_channel->recog_request->pool);
    if (!message) {
        return FALSE;
    }

    /* set request state */
    message->start_line.request_state = MRCP_REQUEST_STATE_INPROGRESS;
    /* send asynch event */
    return mrcp_engine_channel_message_send(recog_channel->channel, message);
}

/* Load demo recognition result */
static apt_bool_t huawei_recog_result_load(huawei_recog_channel_t *recog_channel, mrcp_message_t *message)
{
    mrcp_engine_channel_t *channel = recog_channel->channel;
    const apt_dir_layout_t *dir_layout = channel->engine->dir_layout;

    /* read the demo result from file  这里是从asr真实结果中读取*/
    mrcp_generic_header_t *generic_header;

    apt_string_assign_n(&message->body, recog_channel->call_back->Result().c_str(), recog_channel->call_back->Result().size(), message->pool);

    /* get/allocate generic header */
    generic_header = mrcp_generic_header_prepare(message);
    if (generic_header) {
        /* set content types */
        apt_string_assign(&generic_header->content_type, "application/x-nlsml", message->pool);
        mrcp_generic_header_property_add(message, GENERIC_HEADER_CONTENT_TYPE);
    }
    return TRUE;
}

/* Raise huawei RECOGNITION-COMPLETE event */
static apt_bool_t huawei_recog_recognition_complete(
    huawei_recog_channel_t *recog_channel, mrcp_recog_completion_cause_e cause)
{
    mrcp_recog_header_t *recog_header;
    /* create RECOGNITION-COMPLETE event */
    mrcp_message_t *message = mrcp_event_create(
        recog_channel->recog_request, RECOGNIZER_RECOGNITION_COMPLETE, recog_channel->recog_request->pool);
    if (!message) {
        return FALSE;
    }

    /* get/allocate recognizer header */
    recog_header = (mrcp_recog_header_t *)mrcp_resource_header_prepare(message);
    if (recog_header) {
        /* set completion cause */
        recog_header->completion_cause = cause;
        mrcp_resource_header_property_add(message, RECOGNIZER_HEADER_COMPLETION_CAUSE);
    }
    /* set request state */
    message->start_line.request_state = MRCP_REQUEST_STATE_COMPLETE;

    if (cause == RECOGNIZER_COMPLETION_CAUSE_SUCCESS) {
        huawei_recog_result_load(recog_channel, message);
    }

    recog_channel->recog_request = NULL;
    /* send asynch event */
    return mrcp_engine_channel_message_send(recog_channel->channel, message);
}

/** Callback is called from MPF engine context to write/send new frame */
static apt_bool_t huawei_recog_stream_write(mpf_audio_stream_t *stream, const mpf_frame_t *frame)
{
    huawei_recog_channel_t *recog_channel = (huawei_recog_channel_t *)stream->obj;
    if (recog_channel->stop_response) {
        /* send asynchronous response to STOP request */
        mrcp_engine_channel_message_send(recog_channel->channel, recog_channel->stop_response);
        recog_channel->stop_response = NULL;
        recog_channel->recog_request = NULL;
        return TRUE;
    }
    bool finish = false;

    // 接收的数据帧转给SDK
    if (recog_channel->recog_request) {
        // 第622行至660行 是unimrcp提供的断句规则,这里禁用中间两个case(因为效果差),直接用华为云的断句规则
        // mpf_detector_event_e det_event = mpf_activity_detector_process(recog_channel->detector, frame);
        // switch (det_event) {
        //     case MPF_DETECTOR_EVENT_ACTIVITY:
        //         apt_log(RECOG_LOG_MARK,
        //             APT_PRIO_INFO,
        //             "Detected Voice Activity " APT_SIDRES_FMT,
        //             MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
        //         huawei_recog_start_of_input(recog_channel);
        //         break;
        //     case MPF_DETECTOR_EVENT_INACTIVITY:
        //         apt_log(RECOG_LOG_MARK,
        //             APT_PRIO_INFO,
        //             "Detected Voice Inactivity " APT_SIDRES_FMT,
        //             MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
        //         if (recog_channel->rasr_client) {
        //             recog_channel->rasr_client->SendEnd();
        //             recog_channel->rasr_client->Close();
        //         }
        //         finish = true;
        //         huawei_recog_recognition_complete(recog_channel, RECOGNIZER_COMPLETION_CAUSE_SUCCESS);
        //         apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "get RECOGNIZER_COMPLETION_CAUSE_SUCCESS stop recog");
        //         break;
        //     case MPF_DETECTOR_EVENT_NOINPUT:
        //         apt_log(RECOG_LOG_MARK,
        //             APT_PRIO_INFO,
        //             "Detected Noinput " APT_SIDRES_FMT,
        //             MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
        //         if (recog_channel->timers_started == TRUE) {
        //             if (recog_channel->rasr_client) {
        //                 recog_channel->rasr_client->SendEnd();
        //                 recog_channel->rasr_client->Close();
        //             }
        //             huawei_recog_recognition_complete(recog_channel,RECOGNIZER_COMPLETION_CAUSE_NO_INPUT_TIMEOUT);
        //         }
        //         finish = true;
        //         break;
        //     default:
        //         break;
        // }
        Status cur_status = recog_channel->call_back->GetStatus();
        switch (cur_status) {
            case Status::voice_start:
                apt_log(RECOG_LOG_MARK,
                    APT_PRIO_INFO,
                    "Detected Voice Activity " APT_SIDRES_FMT,
                    MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
                huawei_recog_start_of_input(recog_channel);
                recog_channel->call_back->SetStatus(Status::recognizing);
                break;
            case Status::voice_end:
                apt_log(RECOG_LOG_MARK,
                    APT_PRIO_INFO,
                    "Detected Voice Inactivity " APT_SIDRES_FMT,
                    MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
                if (recog_channel->rasr_client) {
                    recog_channel->rasr_client->SendEnd();
                    recog_channel->rasr_client->Close();
                }
                finish = true;
                huawei_recog_recognition_complete(recog_channel, RECOGNIZER_COMPLETION_CAUSE_SUCCESS);
                apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "get RECOGNIZER_COMPLETION_CAUSE_SUCCESS stop recog");
                break;
            case Status::exceeded_silence:
                apt_log(RECOG_LOG_MARK,
                    APT_PRIO_INFO,
                    "Detected Noinput " APT_SIDRES_FMT,
                    MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
                if (recog_channel->timers_started == TRUE) {
                    if (recog_channel->rasr_client) {
                        recog_channel->rasr_client->SendEnd();
                        recog_channel->rasr_client->Close();
                    }
                    huawei_recog_recognition_complete(recog_channel,RECOGNIZER_COMPLETION_CAUSE_NO_INPUT_TIMEOUT);
                }
                finish = true;
                break;
            default:
                break;
        }        

        if (recog_channel->recog_request) {
            if ((frame->type & MEDIA_FRAME_TYPE_EVENT) == MEDIA_FRAME_TYPE_EVENT) {
                if (frame->marker == MPF_MARKER_START_OF_EVENT) {
                    apt_log(RECOG_LOG_MARK,
                        APT_PRIO_INFO,
                        "Detected Start of Event " APT_SIDRES_FMT " id:%d",
                        MRCP_MESSAGE_SIDRES(recog_channel->recog_request),
                        frame->event_frame.event_id);
                } else if (frame->marker == MPF_MARKER_END_OF_EVENT) {
                    apt_log(RECOG_LOG_MARK,
                        APT_PRIO_INFO,
                        "Detected End of Event " APT_SIDRES_FMT " id:%d duration:%d ts",
                        MRCP_MESSAGE_SIDRES(recog_channel->recog_request),
                        frame->event_frame.event_id,
                        frame->event_frame.duration);
                }
            }
        }

        if (recog_channel->audio_out) {
            fwrite(frame->codec_frame.buffer, 1, frame->codec_frame.size, recog_channel->audio_out);
        }

        if (recog_channel->rasr_client && !finish) {
            recog_channel->rasr_client->SendBinary((unsigned char*)frame->codec_frame.buffer, frame->codec_frame.size);
            apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "Sending [%d] bytes", frame->codec_frame.size);
        }
    }
    return TRUE;
}

static apt_bool_t huawei_recog_msg_signal(
    huawei_recog_msg_type_e type, mrcp_engine_channel_t *channel, mrcp_message_t *request)
{
    apt_bool_t status = FALSE;
    huawei_recog_channel_t *huawei_channel = (huawei_recog_channel_t *)channel->method_obj;
    huawei_recog_engine_t *huawei_engine = huawei_channel->huawei_engine;
    apt_task_t *task = apt_consumer_task_base_get(huawei_engine->task);
    apt_task_msg_t *msg = apt_task_msg_get(task);
    if (msg) {
        huawei_recog_msg_t *huawei_msg;
        msg->type = TASK_MSG_USER;
        huawei_msg = (huawei_recog_msg_t *)msg->data;

        huawei_msg->type = type;
        huawei_msg->channel = channel;
        huawei_msg->request = request;
        status = apt_task_msg_signal(task, msg);
    }
    return status;
}

static apt_bool_t huawei_recog_msg_process(apt_task_t *task, apt_task_msg_t *msg)
{
    huawei_recog_msg_t *huawei_msg = (huawei_recog_msg_t *)msg->data;
    switch (huawei_msg->type) {
        case HUAWEI_RECOG_MSG_OPEN_CHANNEL:
            /* open channel and send asynch response */
            mrcp_engine_channel_open_respond(huawei_msg->channel, TRUE);
            apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "HUAWEI_RECOG_MSG_OPEN_CHANNEL");
            break;
        case HUAWEI_RECOG_MSG_CLOSE_CHANNEL: {
            /* close channel, make sure there is no activity and send asynch response */
            huawei_recog_channel_t *recog_channel = (huawei_recog_channel_t *)huawei_msg->channel->method_obj;
            if (recog_channel->audio_out) {
                fclose(recog_channel->audio_out);
                recog_channel->audio_out = NULL;
            }
            apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "HUAWEI_RECOG_MSG_CLOSE_CHANNEL");
            mrcp_engine_channel_close_respond(huawei_msg->channel);
            break;
        }
        case HUAWEI_RECOG_MSG_REQUEST_PROCESS:
            apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "HUAWEI_RECOG_MSG_REQUEST_PROCESS");
            huawei_recog_channel_request_dispatch(huawei_msg->channel, huawei_msg->request);
            break;
        default:
            break;
    }
    return TRUE;
}

  1. 修改asr插件的makefile.am文件(进入路径:unimrcp/plugins): vim huawei-recog/Makefile.am

AM_CPPFLAGS                = $(UNIMRCP_PLUGIN_INCLUDES)

plugin_LTLIBRARIES         = huaweirecog.la

huaweirecog_la_SOURCES       = src/huawei_recog_engine.cc
huaweirecog_la_LDFLAGS       = $(UNIMRCP_PLUGIN_OPTS) \
                               -L$(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party \
                               -L$(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/jsoncpp-install/lib64 \
                               -L$(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/gflags-install/lib    \
                               -L$(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/glog-install/lib64    \
                               ${CMAKE_SOURCE_DIR}/third_party/openssl-install/lib                                    \
                               -lhuawei_rasr -ssl -lcrypto -ljsoncpp -lgflags_nothreads -stdc++=14
huaweirecog_ladir            = $(libdir)
huaweirecog_la_DATA          = $(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/libhuawei_rasr.so       \
                               $(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/openssl-install/lib/libcrypto.so  \
                               $(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/openssl-install/lib/libssl.so     \
                               $(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/jsoncpp-install/lib64/libjsoncpp.so \
                               $(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/gflags-install/lib/libgflags_nothreads.so


include $(top_srcdir)/build/rules/uniplugin.am

UNIMRCP_PLUGIN_INCLUDES      += -I$(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/include  \
                                -I$(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/websocketpp-src \
                                -I$(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/jsoncpp-install/include     \
                                -I$(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/boost-src/                 \
                                -I$(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/openssl-install/include    \
                                -I$(top_srcdir)/plugins/huawei-recog/huaweicloud_asr/third_party/gflags-install/include

  1. 修改插件的Makefile.am(进入路径:unimrcp/plugins) vim Makefile.am
MAINTAINERCLEANFILES   = Makefile.in

SUBDIRS                =

if DEMOSYNTH_PLUGIN
SUBDIRS               += demo-synth
endif

if HUAWEIRECOG_PLUGIN
SUBDIRS               += huawei-recog
endif

if DEMOVERIFIER_PLUGIN
SUBDIRS               += demo-verifier
endif

if RECORDER_PLUGIN
SUBDIRS               += mrcp-recorder
endif
  1. 修改最顶层的configure.ac(进入unimrcp根目录):
dnl
dnl Autoconf configuration file for UniMRCP.
dnl
dnl Use ./bootstrap to produce a configure script.
dnl

AC_PREREQ(2.59)

AC_INIT([unimrcp],[1.8.0])

AC_CONFIG_AUX_DIR([build])
AC_CONFIG_MACRO_DIR([build/acmacros])

dnl Set ac_macro_dir variable manually for autoconf 2.61 and above.
ac_macro_dir="build/acmacros"

AC_SUBST(ac_aux_dir)
AC_SUBST(ac_macro_dir)

dnl Include m4 macros for libtool 2.
sinclude(build/acmacros/libtool.m4)
sinclude(build/acmacros/ltoptions.m4)
sinclude(build/acmacros/ltsugar.m4)
sinclude(build/acmacros/ltversion.m4)
sinclude(build/acmacros/lt~obsolete.m4)

AC_PREFIX_DEFAULT(/usr/local/unimrcp)

dnl Define the directory layout.
APR_ENABLE_LAYOUT(classic, [plugindir logdir vardir])
AC_SUBST(plugindir)
AC_SUBST(logdir)
AC_SUBST(vardir)

dnl Reparse the configure arguments.
APR_PARSE_ARGUMENTS

dnl Generate ./config.nice to reuse ./configure command-line.
APR_CONFIG_NICE(config.nice)

AM_INIT_AUTOMAKE([no-define nostdinc foreign subdir-objects])

dnl Enable silent build rules available since automake 1.11.
m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])])

dnl Set default language.
AC_LANG_C

AC_PROG_CC
AC_PROG_CXX
AM_PROG_CC_C_O
AC_PROG_INSTALL

dnl Skip detection of Fortran.
m4_undefine([AC_PROG_F77])
m4_defun([AC_PROG_F77],[])
AC_PROG_LIBTOOL

dnl Suppress warning: ar: 'u' modifier ignored since 'D' is the default
AC_SUBST(AR_FLAGS, [cr])

dnl Do not use autoconf generated compiler DEFS.
rm confdefs.h
touch confdefs.h

dnl Check for C compiler vendor.
AX_COMPILER_VENDOR

dnl Search for pkg-config.
AC_PATH_PROG(PKG_CONFIG, pkg-config, no)
if test "x$PKG_CONFIG" = "xno"; then
    AC_MSG_ERROR([pkg-config is missing])
fi

dnl Get version information.
get_version="$srcdir/build/get-version.sh"
version_hdr="$srcdir/build/uni_version.h"
plugin_version_hdr="$srcdir/libs/mrcp-engine/include/mrcp_engine_plugin.h"
UNI_DOTTED_VERSION="`$get_version all $version_hdr UNI`"
UNI_LT_VERSION="-version-info `$get_version libtool $version_hdr UNI`"
PLUGIN_LT_VERSION="-version-info `$get_version libtool $plugin_version_hdr PLUGIN`"

AC_SUBST(UNI_DOTTED_VERSION)
AC_SUBST(UNI_LT_VERSION)
AC_SUBST(PLUGIN_LT_VERSION)

echo "UniMRCP Version: ${UNI_DOTTED_VERSION}"

dnl Check for the APR and APR-util libraries.
UNIMRCP_CHECK_APR
dnl Check for the Sofia-SIP library.
UNIMRCP_CHECK_SOFIA

dnl Enable AMR codec.
AC_ARG_ENABLE(amr-codec,
    [AC_HELP_STRING([--enable-amr-codec  ],[enable AMR codec])],
    [enable_amr_codec="$enableval"],
    [enable_amr_codec="no"])

AC_MSG_NOTICE([enable AMR codec: $enable_amr_codec])
if test "${enable_amr_codec}" != "no"; then
    dnl Check for the OpenCORE AMR library.
    UNIMRCP_CHECK_OPENCORE_AMR

    dnl Check for the VO AMRWBENC library.
    UNIMRCP_CHECK_VO_AMRWBENC
fi

AM_CONDITIONAL([UNIMRCP_AMR_CODEC],[test "${enable_amr_codec}" != "no"])

dnl Enable inter-library dependencies.
AC_ARG_ENABLE(interlib-deps,
    [AC_HELP_STRING([--disable-interlib-deps  ],[disable inter-library dependencies (might break builds)])],
    [enable_interlib_deps="$enableval"],
    [enable_interlib_deps="yes"])

AC_MSG_NOTICE([enable inter-library dependencies: $enable_interlib_deps])
if test "${enable_interlib_deps}" == "yes"; then
    link_all_deplibs=yes
    link_all_deplibs_CXX=yes
else
    link_all_deplibs=no
    link_all_deplibs_CXX=no
fi

dnl Enable maintainer mode.
AC_ARG_ENABLE(maintainer-mode,
    [AC_HELP_STRING([--enable-maintainer-mode  ],[turn on debugging and compile time warnings])],
    [enable_maintainer_mode="$enableval"],
    [enable_maintainer_mode="no"])

AC_MSG_NOTICE([enable maintainer mode: $enable_maintainer_mode])
if test "${enable_maintainer_mode}" != "no"; then
    APR_ADDTO(CFLAGS,-g)
    if test "x${ax_cv_c_compiler_vendor}"  =  "xgnu" ; then
        APR_ADDTO(CFLAGS,-Wall -Werror)
    fi
fi

dnl UniMRCP client library.
AC_ARG_ENABLE(client-lib,
    [AC_HELP_STRING([--disable-client-lib  ],[exclude unimrcpclient lib from build])],
    [enable_client_lib="$enableval"],
    [enable_client_lib="yes"])

AM_CONDITIONAL([UNIMRCP_CLIENT_LIB],[test "${enable_client_lib}" = "yes"])

dnl Sample UniMRCP client application in C.
AC_ARG_ENABLE(client-app,
    [AC_HELP_STRING([--disable-client-app  ],[exclude sample unimrcpclient app from build])],
    [enable_client_app="$enableval"],
    [enable_client_app="yes"])

AM_CONDITIONAL([UNIMRCP_CLIENT_APP],[test "${enable_client_lib}" = "yes" && test "${enable_client_app}" = "yes"])

dnl Sample UniMRCP client application in C++.
AC_ARG_ENABLE(umc,
    [AC_HELP_STRING([--disable-umc         ],[exclude sample unimrcpclient C++ app from build])],
    [enable_umc="$enableval"],
    [enable_umc="yes"])

AM_CONDITIONAL([UMC],[test "${enable_client_lib}" = "yes" && test "${enable_umc}" = "yes"])

dnl Miscellaneous ASR client library and application.
AC_ARG_ENABLE(asr-client,
    [AC_HELP_STRING([--disable-asr-client  ],[exclude misc ASR client lib and app from build])],
    [enable_asr_client="$enableval"],
    [enable_asr_client="yes"])

AM_CONDITIONAL([ASR_CLIENT],[test "${enable_client_lib}" = "yes" && test "${enable_asr_client}" = "yes"])

AM_CONDITIONAL([COMMON_CLIENT_DATA],[test "${enable_client_app}" = "yes" || test "${enable_umc}" = "yes" ||test "${enable_asr_client}" = "yes"])

dnl UniMRCP server library.
AC_ARG_ENABLE(server-lib,
    [AC_HELP_STRING([--disable-server-lib  ],[exclude unimrcpserver lib from build])],
    [enable_server_lib="$enableval"],
    [enable_server_lib="yes"])

AM_CONDITIONAL([UNIMRCP_SERVER_LIB],[test "${enable_server_lib}" = "yes"])

dnl UniMRCP server application.
AC_ARG_ENABLE(server-app,
    [AC_HELP_STRING([--disable-server-app  ],[exclude unimrcpserver app from build])],
    [enable_server_app="$enableval"],
    [enable_server_app="yes"])

AM_CONDITIONAL([UNIMRCP_SERVER_APP],[test "${enable_server_lib}" = "yes" && test "${enable_server_app}" = "yes"])

dnl Demo synthesizer plugin.
UNI_PLUGIN_ENABLED(demosynth)

AM_CONDITIONAL([DEMOSYNTH_PLUGIN],[test "${enable_demosynth_plugin}" = "yes"])

dnl Huawei recognizer plugin.
UNI_PLUGIN_ENABLED(huaweirecog)

AM_CONDITIONAL([HUAWEIRECOG_PLUGIN],[test "${enable_huaweirecog_plugin}" = "yes"])

dnl Demo verifier plugin.
UNI_PLUGIN_ENABLED(demoverifier)

AM_CONDITIONAL([DEMOVERIFIER_PLUGIN],[test "${enable_demoverifier_plugin}" = "yes"])

dnl Recorder plugin.
UNI_PLUGIN_ENABLED(recorder)

AM_CONDITIONAL([RECORDER_PLUGIN],[test "${enable_recorder_plugin}" = "yes"])

dnl Enable test suites.
AC_ARG_ENABLE(test-suites,
    [AC_HELP_STRING([--enable-test-suites  ],[build test suites])],
    [enable_test_suites="$enableval"],
    [enable_test_suites="no"])

AM_CONDITIONAL([TEST_SUITES],[test "${enable_test_suites}" != "no"])

AM_CONDITIONAL(ISMAC, [test `uname -s` = Darwin])

AC_CONFIG_FILES([
    Makefile
    libs/Makefile
    libs/apr-toolkit/Makefile
    libs/mpf/Makefile
    libs/mrcp/Makefile
    libs/mrcp-signaling/Makefile
    libs/mrcpv2-transport/Makefile
    libs/mrcp-engine/Makefile
    libs/mrcp-server/Makefile
    libs/mrcp-client/Makefile
    libs/uni-rtsp/Makefile
    modules/Makefile
    modules/mrcp-sofiasip/Makefile
    modules/mrcp-unirtsp/Makefile
    plugins/Makefile
    plugins/mrcp-recorder/Makefile
    plugins/demo-synth/Makefile
    plugins/huawei-recog/Makefile
    plugins/demo-verifier/Makefile
    platforms/Makefile
    platforms/libunimrcp-server/Makefile
    platforms/libunimrcp-client/Makefile
    platforms/unimrcp-server/Makefile
    platforms/unimrcp-client/Makefile
    platforms/libasr-client/Makefile
    platforms/asr-client/Makefile
    platforms/umc/Makefile
    tests/Makefile
    tests/apttest/Makefile
    tests/mpftest/Makefile
    tests/mrcptest/Makefile
    tests/rtsptest/Makefile
    tests/strtablegen/Makefile
    build/Makefile
    build/pkgconfig/Makefile
    build/pkgconfig/unimrcpclient.pc
    build/pkgconfig/unimrcpserver.pc
    build/pkgconfig/unimrcpplugin.pc
    conf/Makefile
    data/Makefile
    docs/doxygen.conf
])

AC_OUTPUT

echo
echo '****************************** REPORT ******************************'
echo
echo UniMRCP version............... : $UNI_DOTTED_VERSION
echo
echo APR version................... : $apr_version
echo APR-util version.............. : $apu_version
echo Sofia-SIP version............. : $sofia_version
if test "${enable_amr_codec}" != "no"; then
echo
echo OpenCORE AMR version.......... : $opencore_amr_version
echo VO AMRWBENC version........... : $vo_amrwbenc_version
fi
echo
echo Compiler...................... : $CC
echo Compiler flags................ : $CFLAGS
echo Preprocessor definitions...... : $CPPFLAGS
echo Linker flags.................. : $LDFLAGS
echo
echo UniMRCP client lib............ : $enable_client_lib
echo Sample UniMRCP client app..... : $enable_client_app
echo Sample UMC C++ client app..... : $enable_umc
echo Misc ASR client lib and app... : $enable_asr_client
echo
echo UniMRCP server lib............ : $enable_server_lib
echo UniMRCP server app............ : $enable_server_app
echo
echo Demo synthesizer plugin....... : $enable_demosynth_plugin
echo Huawei recognizer plugin........ : $enable_huaweirecog_plugin
echo Demo verifier plugin.......... : $enable_demoverifier_plugin
echo Recorder plugin............... : $enable_recorder_plugin
echo
echo Installation layout........... : $layout_name
echo Installation directory........ : $prefix
echo
echo '********************************************************************'

  1. 添加asr配置文件(进入unimrcp根目录),这些都是必填项,按照备注获取相应的asr鉴权信息:vim conf/asr_info.conf
# aksk是鉴权信息,登录console控制台,进入我的凭证获取
--ak=
--sk=
# 华为云ASR提供北京四和上海一两个局点,客户自行选择
--region=cn-north-4
# projectID要与对应的region相对应
--projectId=
--endpoint=sis-ext.cn-north-4.myhuaweicloud.com
--audioFormat=pcm8k16bit
# 16k推荐使用chinese_16k_general, 8k推荐使用chinese_8k_general
--property=chinese_8k_general
# 华为云实时语音识别提供三种模式,分别是continue/short/sentence; 由于mrcp默认有自己的断句逻辑,建议用short模式
--asr_mode=sentence
--vadHead=10000
--vadTail=1500
--maxSeconds=60
  1. 重新编译unimrcp,默认会安装在/usr/local/unimrcp中
./bootstrap
make -j
make install
  1. unimrcp代码结构展示

image.png

/usr/local/unimrcp
├── bin                 # 可执行文件 
├── conf                # 配置文件,可以设置访问IP和端口号
├── data                # 默认提供的一些音频文件
├── include             # 程序必须的头文件
├── lib                 # 依赖的库文件
├── log                 # 日志路径
├── plugin              # 插件所依赖的库文件
└── var                 # 保存的音频文件

  1. 运行结果展示

启动服务端 /usr/local/unimrcp/bin/unimrcpserver

(base) [root@asr-build bin]# ./unimrcpserver 
2024-03-11 17:17:14:704844 [NOTICE] UniMRCP Server [1.8.0]
2024-03-11 17:17:14:704913 [INFO]   APR [1.5.2]
2024-03-11 17:17:14:704924 [NOTICE] Create MRCP Server
2024-03-11 17:17:14:704946 [NOTICE] Open Config File [/usr/local/unimrcp/conf/unimrcpserver.xml]
2024-03-11 17:17:14:705077 [INFO]   Set Property ip:127.0.0.1
2024-03-11 17:17:14:705086 [INFO]   Register Codec [PCMU]
2024-03-11 17:17:14:705088 [INFO]   Register Codec [PCMA]
2024-03-11 17:17:14:705090 [INFO]   Register Codec [L16]
2024-03-11 17:17:14:705092 [INFO]   Register Codec [G722]
2024-03-11 17:17:14:705127 [NOTICE] Register Resource [speechsynth]
2024-03-11 17:17:14:705132 [NOTICE] Register Resource [speechrecog]
2024-03-11 17:17:14:705135 [NOTICE] Register Resource [recorder]
2024-03-11 17:17:14:705138 [NOTICE] Register Resource [speakverify]
2024-03-11 17:17:14:705141 [INFO]   Register Resource Factory
2024-03-11 17:17:14:705157 [NOTICE] Create SofiaSIP Agent [SIP-Agent-1] [1.12.11-239-g54ef3e2] sip:127.0.0.1:8060;transport=udp,tcp
2024-03-11 17:17:14:705163 [INFO]   Register Signaling Agent [SIP-Agent-1]
2024-03-11 17:17:14:705178 [NOTICE] Create RTSP Server [RTSP-Agent-1] 127.0.0.1:1554 [100] connection timeout [600 sec]
2024-03-11 17:17:14:705258 [INFO]   Register Signaling Agent [RTSP-Agent-1]
2024-03-11 17:17:14:705266 [NOTICE] Create MRCPv2 Agent [MRCPv2-Agent-1] 127.0.0.1:1544 [100]
2024-03-11 17:17:14:705290 [INFO]   Register Connection Agent [MRCPv2-Agent-1]
2024-03-11 17:17:14:705297 [NOTICE] Create Media Engine [Media-Engine-1]
2024-03-11 17:17:14:705305 [INFO]   Register Media Engine [Media-Engine-1]
2024-03-11 17:17:14:705309 [NOTICE] Create RTP Termination Factory 127.0.0.1:[5000,6000]
2024-03-11 17:17:14:705311 [INFO]   Register RTP Termination Factory [RTP-Factory-1]
2024-03-11 17:17:14:705316 [INFO]   Load Plugin [Demo-Synth-1] [/usr/local/unimrcp/plugin/demosynth.so]
2024-03-11 17:17:14:705393 [INFO]   Register MRCP Engine [Demo-Synth-1]
2024-03-11 17:17:14:705397 [INFO]   Load Plugin [Huawei-Recog-1] [/usr/local/unimrcp/plugin/huaweirecog.so]
2024-03-11 17:17:14:712580 [INFO]   Register MRCP Engine [Huawei-Recog-1]
2024-03-11 17:17:14:712599 [INFO]   Load Plugin [Demo-Verifier-1] [/usr/local/unimrcp/plugin/demoverifier.so]
2024-03-11 17:17:14:712677 [INFO]   Register MRCP Engine [Demo-Verifier-1]
2024-03-11 17:17:14:712682 [INFO]   Load Plugin [Recorder-1] [/usr/local/unimrcp/plugin/mrcprecorder.so]
2024-03-11 17:17:14:712739 [INFO]   Register MRCP Engine [Recorder-1]
2024-03-11 17:17:14:712755 [INFO]   Register RTP Settings [RTP-Settings-1]
2024-03-11 17:17:14:712764 [NOTICE] Create MRCPv2 Profile [uni2]
2024-03-11 17:17:14:712773 [INFO]   Associate Resource [speechsynth] to Engine [Demo-Synth-1] in Profile [uni2]
2024-03-11 17:17:14:712776 [INFO]   Associate Resource [speechrecog] to Engine [Huawei-Recog-1] in Profile [uni2]
2024-03-11 17:17:14:712779 [INFO]   Associate Resource [recorder] to Engine [Recorder-1] in Profile [uni2]
2024-03-11 17:17:14:712782 [INFO]   Associate Resource [speakverify] to Engine [Demo-Verifier-1] in Profile [uni2]
2024-03-11 17:17:14:712786 [INFO]   Register Profile [uni2]
2024-03-11 17:17:14:712790 [NOTICE] Create MRCPv1 Profile [uni1]
2024-03-11 17:17:14:712793 [INFO]   Associate Resource [speechsynth] to Engine [Demo-Synth-1] in Profile [uni1]
2024-03-11 17:17:14:712796 [INFO]   Associate Resource [speechrecog] to Engine [Huawei-Recog-1] in Profile [uni1]
2024-03-11 17:17:14:712799 [INFO]   Associate Resource [recorder] to Engine [Recorder-1] in Profile [uni1]
2024-03-11 17:17:14:712802 [INFO]   Associate Resource [speakverify] to Engine [Demo-Verifier-1] in Profile [uni1]
2024-03-11 17:17:14:712805 [INFO]   Register Profile [uni1]
2024-03-11 17:17:14:712810 [INFO]   Start Task [MRCP Server]
>2024-03-11 17:17:14:712891 [INFO]   Open Engine [Demo-Verifier-1]
2024-03-11 17:17:14:712906 [INFO]   Start Task [Demo Verifier Engine]
2024-03-11 17:17:14:712963 [INFO]   Open Engine [Huawei-Recog-1]
2024-03-11 17:17:14:712966 [INFO]   Start Task [Huawei Recog Engine]
2024-03-11 17:17:14:712990 [INFO]   Open Engine [Recorder-1]
2024-03-11 17:17:14:712993 [INFO]   Open Engine [Demo-Synth-1]
2024-03-11 17:17:14:712995 [INFO]   Start Task [Demo Synth Engine]
2024-03-11 17:17:14:713019 [INFO]   Start Task [SIP-Agent-1]
2024-03-11 17:17:14:713037 [INFO]   Start Task [RTSP-Agent-1]
2024-03-11 17:17:14:713159 [INFO]   Start Task [MRCPv2-Agent-1]
2024-03-11 17:17:14:713313 [INFO]   Start Task [Media-Engine-1]
sres: /etc/resolv.conf: unknown option                      
2024-03-11 17:17:14:714108 [NOTICE] MRCP Server Started

启动客户端:/usr/local/unimrcp/umc,然后在交互窗口执行run recog,获取识别结果

image.png

四、unimrcp定制化改动

1. 修改xml文件(在conf文件夹中),指定访问地址和端口号。端口号我们采用默认端口,IP地址指定为本地IP

unimrcpclient.xml修改:

image.png

unimrcpserver.xml修改:

image.png

2. 利用unimrcp调整ASR断句参数

unimrcp的header里面提供了一组断句参数,来控制响应时间/句子长段(比如,说完一段话之后,等待多少毫秒,给出结果)。各参数名及其含义如下(在路径data/params_default.txt下)

No-Input-Timeout: 5000       -----> 用户接到电话后超过5000ms没有声音,就结束识别 等价于华为云ASR的vad_head
Recognition-Timeout: 20000    ------> 用户一直说话,最长能说多久(默认20s),结束识别。等价于华为云ASR的max_seconds
Speech-Complete-Timeout: 400  -------> 用户开始说话后,停顿多久(默认400ms)就结束说话
2.1 通过请求参数传递断句参数

以上参数均是通过请求体的header进行传递,参考文档:https://www.unimrcp.org/manuals/html/GoogleSRUsageManual.html

可以通过unimrcp打印的日志,查看参数设置是否生效:

image.png

2.2 强制指定断句参数

为了方便管理,建议对所有通话保留相同的断句参数,具体设置如下:

	if(recog_header) {
		if(mrcp_resource_header_property_check(request,RECOGNIZER_HEADER_START_INPUT_TIMERS) == TRUE) {
			recog_channel->timers_started = recog_header->start_input_timers;
		}
		if(mrcp_resource_header_property_check(request,RECOGNIZER_HEADER_NO_INPUT_TIMEOUT) == TRUE) {
			mpf_activity_detector_noinput_timeout_set(recog_channel->detector,10000); // 强制指定 如果10s中无人说话,就结束
		}
		if(mrcp_resource_header_property_check(request,RECOGNIZER_HEADER_SPEECH_COMPLETE_TIMEOUT) == TRUE) {
			mpf_activity_detector_silence_timeout_set(recog_channel->detector, 600); // 强制指定600ms
		}
	}

3. 利用华为云ASR做智能短句

华为云ASR提供的sentence模式,可适用于外呼、数字人场景,通过设置vad_head/vad_tail/max_seconds参数可以智能短句,相关代码变动如下

    if (recog_channel->recog_request) {
        // 第622行至660行 是unimrcp提供的断句规则,这里禁用中间两个case(因为效果差),直接用华为云的断句规则
        // mpf_detector_event_e det_event = mpf_activity_detector_process(recog_channel->detector, frame);
        // switch (det_event) {
        //     case MPF_DETECTOR_EVENT_ACTIVITY:
        //         apt_log(RECOG_LOG_MARK,
        //             APT_PRIO_INFO,
        //             "Detected Voice Activity " APT_SIDRES_FMT,
        //             MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
        //         huawei_recog_start_of_input(recog_channel);
        //         break;
        //     case MPF_DETECTOR_EVENT_INACTIVITY:
        //         apt_log(RECOG_LOG_MARK,
        //             APT_PRIO_INFO,
        //             "Detected Voice Inactivity " APT_SIDRES_FMT,
        //             MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
        //         if (recog_channel->rasr_client) {
        //             recog_channel->rasr_client->SendEnd();
        //             recog_channel->rasr_client->Close();
        //         }
        //         finish = true;
        //         huawei_recog_recognition_complete(recog_channel, RECOGNIZER_COMPLETION_CAUSE_SUCCESS);
        //         apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "get RECOGNIZER_COMPLETION_CAUSE_SUCCESS stop recog");
        //         break;
        //     case MPF_DETECTOR_EVENT_NOINPUT:
        //         apt_log(RECOG_LOG_MARK,
        //             APT_PRIO_INFO,
        //             "Detected Noinput " APT_SIDRES_FMT,
        //             MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
        //         if (recog_channel->timers_started == TRUE) {
        //             if (recog_channel->rasr_client) {
        //                 recog_channel->rasr_client->SendEnd();
        //                 recog_channel->rasr_client->Close();
        //             }
        //             huawei_recog_recognition_complete(recog_channel,RECOGNIZER_COMPLETION_CAUSE_NO_INPUT_TIMEOUT);
        //         }
        //         finish = true;
        //         break;
        //     default:
        //         break;
        // }
        Status cur_status = recog_channel->call_back->GetStatus();
        switch (cur_status) {
            case Status::voice_start:
                apt_log(RECOG_LOG_MARK,
                    APT_PRIO_INFO,
                    "Detected Voice Activity " APT_SIDRES_FMT,
                    MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
                huawei_recog_start_of_input(recog_channel);
                recog_channel->call_back->SetStatus(Status::recognizing);
                break;
            case Status::voice_end:
                apt_log(RECOG_LOG_MARK,
                    APT_PRIO_INFO,
                    "Detected Voice Inactivity " APT_SIDRES_FMT,
                    MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
                if (recog_channel->rasr_client) {
                    recog_channel->rasr_client->SendEnd();
                    recog_channel->rasr_client->Close();
                }
                finish = true;
                huawei_recog_recognition_complete(recog_channel, RECOGNIZER_COMPLETION_CAUSE_SUCCESS);
                apt_log(RECOG_LOG_MARK, APT_PRIO_INFO, "get RECOGNIZER_COMPLETION_CAUSE_SUCCESS stop recog");
                break;
            case Status::exceeded_silence:
                apt_log(RECOG_LOG_MARK,
                    APT_PRIO_INFO,
                    "Detected Noinput " APT_SIDRES_FMT,
                    MRCP_MESSAGE_SIDRES(recog_channel->recog_request));
                if (recog_channel->timers_started == TRUE) {
                    if (recog_channel->rasr_client) {
                        recog_channel->rasr_client->SendEnd();
                        recog_channel->rasr_client->Close();
                    }
                    huawei_recog_recognition_complete(recog_channel,RECOGNIZER_COMPLETION_CAUSE_NO_INPUT_TIMEOUT);
                }
                finish = true;
                break;
            default:
                break;
        }    

五、快速搭建unimrcp服务

作者按照前四章逐步执行,并在GitHub和Gitee上发布了修改后的代码,大家可以直接拿来使用。

本代码仓提供两种安装方式供用户选择

方法一: 源码安装

  1. 先下载安装华为云实时语音识别SDK,安装指南参考https://bbs.huaweicloud.com/blogs/392949

  2. 将安装的华为云ASR SDK放入plugins/huawei_recog/huaweicloud_asr 】

  3. 进入根目录/unimcrp-deps-1.6.0,安装安装unimrcp的依赖库

./build-dep-libs.sh
  1. 进入根目录,安装unimrcp
./bootstrap
./configure
make
make install
  1. unimrcp默认安装在/usr/local/unimrcp中,使用前需要手动修改asr_info.conf中的配置项

方法二:直接使用release库

wget https://gitee.com/computervisionlearner/unimrcp_with_huaweicloud_asr/releases/download/v0.0.1/unimrcp-1.8.0_unimrcp-deps-1.6.0_binary.tar.gz
tar -xzvf unimrcp-1.8.0_unimrcp-deps-1.6.0_binary.tar.gz
【版权声明】本文为华为云社区用户原创内容,转载时必须标注文章的来源(华为云社区)、文章链接、文章作者等基本信息, 否则作者和本社区有权追究责任。如果您发现本社区中有涉嫌抄袭的内容,欢迎发送邮件进行举报,并提供相关证据,一经查实,本社区将立刻删除涉嫌侵权内容,举报邮箱: cloudbbs@huaweicloud.com
  • 点赞
  • 收藏
  • 关注作者

评论(0

0/1000
抱歉,系统识别当前为高风险访问,暂不支持该操作

全部回复

上滑加载中

设置昵称

在此一键设置昵称,即可参与社区互动!

*长度不超过10个汉字或20个英文字符,设置后3个月内不可修改。

*长度不超过10个汉字或20个英文字符,设置后3个月内不可修改。