一、基于PaddleSpeech的低复杂度家庭环境音识别

地址： https://challenge.xfyun.cn/topic/info?type=family-environment-2022

项目地址：https://aistudio.baidu.com/aistudio/projectdetail/4470995

1.赛事背景

声音作为一种重要的信息载体，由于其易收集、不受角度和光线的限制等优点，常被用于辅助环境感知和信息决策，故语音控制普遍应用于智能家居系统。智能设备接收并处理环境中的声音信号，通过声音事件识别技术可以侦测判断出环境中的物体与发生的事件，例如婴儿哭泣声、枪声和敲门声等，并能迅速地感知到环境中的变化，例如脚步声由远及近等，系统据此启动相关的智能设备。因此，声音事件识别技术已被用于安防监控、音频内容检索等智能感知等领域中，为新型的人机交互方式和智能机器听觉系统提供了帮助。

但针对应用侧存在两大主要挑战：1. 数据层面：因环境复杂，含有较多杂音；2. 设备层面：智能家居硬件设备计算力及存储有限。

2.赛事任务

声音识别事件需强大的数据作为支撑，本次大赛提供了品冠科技云平台音频数据作为训练样本，包括6类音频数据：看电视的声音、燃气报警的声音、炒菜的声音、流水的声音、拉窗帘的声音和小孩哭泣的声音，它们的标签分别为1、2、3、4、5、6。音频文件名含有声音类型，参赛者可以据此对文件进行分类。出于数据安全保证的考虑，所有数据均为脱敏处理后的数据。参赛选手需基于提供的样本构建低复杂度量化模型，通过输入音频数据预测声音对应的事件（预测声音的类型）。

本次比赛有模型复杂度限制，模型复杂度以参数量作为度量。参赛选手提交的模型参数量需小于1M，模型参数为量化后INT8形式。模型参数量统计方法统一如下：

https://github.com/AlbertoAncilotto/NeSsi ；量化过程可采用任意量化方法。

二、数据集处理

1.数据集格式处理

!wget https://ai-contest-static.xfyun.cn/2022/%E6%95%B0%E6%8D%AE%E9%9B%86/%E4%BD%8E%E5%A4%8D%E6%9D%82%E5%BA%A6%E5%AE%B6%E5%BA%AD%E7%8E%AF%E5%A2%83%E9%9F%B3%E6%8C%91%E6%88%98%E8%B5%9B%E5%85%AC%E5%BC%80%E6%95%B0%E6%8D%AE.zip -O dataset.zip

--2022-08-29 09:31:26--  https://ai-contest-static.xfyun.cn/2022/%E6%95%B0%E6%8D%AE%E9%9B%86/%E4%BD%8E%E5%A4%8D%E6%9D%82%E5%BA%A6%E5%AE%B6%E5%BA%AD%E7%8E%AF%E5%A2%83%E9%9F%B3%E6%8C%91%E6%88%98%E8%B5%9B%E5%85%AC%E5%BC%80%E6%95%B0%E6%8D%AE.zip
正在解析主机 ai-contest-static.xfyun.cn (ai-contest-static.xfyun.cn)... 220.181.53.219
正在连接 ai-contest-static.xfyun.cn (ai-contest-static.xfyun.cn)|220.181.53.219|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 2361442488 (2.2G) [application/zip]
正在保存至: “dataset.zip”

dataset.zip         100%[===================>]   2.20G  7.34MB/s    in 4m 54s  

2022-08-29 09:36:21 (7.65 MB/s) - 已保存 “dataset.zip” [2361442488/2361442488])

!unzip -qoa -O GBK dataset.zip

!mv 低复杂度家庭环境音挑战赛公开数据 dataset

2.PaddleSpeech安装

!python -m pip install -U -q pip --user
!pip install -q  pytest-runner
!pip install -q  paddlespeech

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
parl 1.4.1 requires pyzmq==18.1.1, but you have pyzmq 23.2.0 which is incompatible.


3.查看声音文件

import warnings
warnings.filterwarnings("ignore")
import IPython
import numpy as np
import matplotlib.pyplot as plt
import paddle
%matplotlib inline

from paddlespeech.audio import load
data, sr = load(file='dataset/train/1_看电视/001.wav', mono=True, dtype='float32')  # 单通道，float32音频样本点
print('wav shape: {}'.format(data.shape))
print('sample rate: {}'.format(sr))

# 展示音频波形
plt.figure()
plt.plot(data)
plt.show()

wav shape: (1920000,)
sample rate: 16000


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2349: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  if isinstance(obj, collections.Iterator):
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2366: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  return list(data) if isinstance(data, collections.MappingView) else data

4.音频文件长度处理

# 查音频长度
import contextlib
import wave
def get_sound_len(file_path):
    with contextlib.closing(wave.open(file_path, 'r')) as f:
        frames = f.getnframes()
        rate = f.getframerate()
        wav_length = frames / float(rate)

    return wav_length

# 编译wav文件
import glob
sound_files=glob.glob('dataset/train/*/*.wav')
print(sound_files[0])
print(len(sound_files))

# 统计最长、最短音频
sounds_len=[]
for sound in sound_files:
    sounds_len.append(get_sound_len(sound))
print("音频最大长度：",max(sounds_len),"秒")
print("音频最小长度：",min(sounds_len),"秒")

dataset/train/3_炒菜/091.wav
616
音频最大长度： 120.0 秒
音频最小长度： 120.0 秒

最长的声音为120秒，现统一尺寸到该长度

!pip install pydub -q

# 音频信息查看
import math
import soundfile as sf
import numpy as np
import librosa

data, samplerate = sf.read('dataset/train/1_看电视/001.wav')
channels = len(data.shape)
length_s = len(data)/float(samplerate)
format_rate=16000
print(f"channels: {channels}")
print(f"length_s: {length_s}")
print(f"samplerate: {samplerate}")

channels: 2
length_s: 120.0
samplerate: 16000

label_list = ['1_看电视',  '2_燃气报警',  '3_炒菜',  '4_流水',  '5_拉窗帘',  '6_小孩哭泣']

# 定义函数，如未达到最大长度，则重复填充，最终从超过34s的音频中截取
from pydub import AudioSegment

def convert_sound_len(filename):
    audio = AudioSegment.from_wav(filename)
    i = 1
    padded = audio*i
    while padded.duration_seconds * 1000 < 120000:
        i = i + 1
        padded = audio * i
    padded[0:120000].set_frame_rate(16000).export(filename, format='wav')

# 统一所有音频到定长
for sound in sound_files:
    convert_sound_len(sound)

5.生成文件列表

按 9：1 生成train和val文件列表

import os
import random

def get_data_list(target_path,train_list_path,eval_list_path):
    '''
    生成数据列表
    '''
    # 获取所有类别保存的文件夹名称
    data_list_path=target_path
    class_dirs = os.listdir(data_list_path)
    if '__MACOSX' in class_dirs:
        class_dirs.remove('__MACOSX')
    # 存储要写进eval.txt和train.txt中的内容
    trainer_list=[]
    eval_list=[]

    #读取每个类别
    ##########################
    random.shuffle(class_dirs)
    ##########################
    for class_dir in class_dirs:
        class_label=label_list.index(class_dir)
        i = 0                
        if class_dir != ".DS_Store":  
            path = os.path.join(data_list_path,class_dir)
            # 获取所有图片
            img_paths = os.listdir(path)
            for img_path in img_paths:                                        # 遍历文件夹下的每个图片
                if img_path =='.DS_Store':
                    continue
                i += 1
                name_path = os.path.join(path,img_path)                       # 每张图片的路径
                if i % 10 == 0:                                                
                    eval_list.append(name_path + ",%d" % class_label + "\n")
                else: 
                    trainer_list.append(name_path + ",%d" % class_label + "\n") 
            class_label += 1

    with open(eval_list_path, 'a') as f:
        for eval_image in eval_list:
            f.write(eval_image) 
    #乱序        
    random.shuffle(trainer_list) 
    with open(train_list_path, 'a') as f2:
        for train_image in trainer_list:
            f2.write(train_image) 
 
    print ('生成数据列表完成！')

target_path="dataset/train"
train_list_path='train_list.csv'
eval_list_path='eval_list.csv'

#每次生成数据列表前，首先清空train_list.csv和eval_list.csv
with open(train_list_path, 'w') as f: 
    f.seek(0)
    f.truncate() 
with open(eval_list_path, 'w') as f: 
    f.seek(0)
    f.truncate() 
#生成数据列表   

get_data_list(target_path,train_list_path,eval_list_path)    

生成数据列表完成！

6.自定义数据集

import os
from paddlespeech.audio.datasets.dataset import AudioClassificationDataset

class CustomDataset(AudioClassificationDataset):

    # 初始化
    def __init__(self, mode, **kwargs):
        files, labels = self._get_data(mode)
        super(CustomDataset, self).__init__(
            files=files, labels=labels, feat_type='raw', **kwargs)

    # 返回音频文件、label值
    def _get_data(self, mode):
        files = []
        labels = []
        file_list=f"{mode}_list.csv"
        with open(file_list,'r') as f:
            lines=f.readlines()
            for line in lines:
                files.append(line.split(',')[0])
                labels.append(line.split(',')[-1])
        return files, labels

# 定义dataloader
import paddle
from paddlespeech.audio.features import LogMelSpectrogram


# Feature config should be align with pretrained model
sample_rate = 16000
feat_conf = {
  'sr': sample_rate,
  'n_fft': 1024,
  'hop_length': 320,
  'window': 'hann',
  'win_length': 1024,
  'f_min': 50.0,
  'f_max': 14000.0,
  'n_mels': 64,
}

feature_extractor = LogMelSpectrogram(**feat_conf)
batch_size=16

train_ds = CustomDataset(mode="train", sample_rate=sample_rate)
train_loader = paddle.io.DataLoader(
    train_ds,
    batch_size=batch_size, 
    shuffle=True)

eval_ds = CustomDataset(mode="eval", sample_rate=sample_rate)
dev_loader = paddle.io.DataLoader(
    eval_ds,
    batch_size=batch_size)   

W0830 11:09:22.568195  6840 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 10.1
W0830 11:09:22.571982  6840 gpu_resources.cc:91] device: 0, cuDNN Version: 7.6.

三、模型训练

1.选取预训练模型

选取cnn14作为 backbone，用于提取音频的特征：

from paddlespeech.cls.models import cnn14
backbone = cnn14(pretrained=True, extract_embedding=True)

[2022-08-30 11:09:23,739] [    INFO] - PaddleAudio | unique_endpoints {''}
[2022-08-30 11:09:23,742] [    INFO] - PaddleAudio | Found /home/aistudio/.paddlespeech/models/panns/panns_cnn14.pdparams

2.构建分类模型

SoundClassifer接收cnn14作为backbone模型，并创建下游的分类网络：

import paddle.nn as nn


class SoundClassifier(nn.Layer):

    def __init__(self, backbone, num_class, dropout=0.1):
        super().__init__()
        self.backbone = backbone
        self.dropout = nn.Dropout(dropout)
        self.fc = nn.Linear(self.backbone.emb_size, num_class)

    def forward(self, x):
        x = x.unsqueeze(1)
        x = self.backbone(x)
        x = self.dropout(x)
        logits = self.fc(x)

        return logits

model = SoundClassifier(backbone, num_class=6)

3.finetune

# 定义优化器和 Loss

optimizer = paddle.optimizer.Adam(learning_rate=1e-4, parameters=model.parameters())
criterion = paddle.nn.loss.CrossEntropyLoss()

from paddlespeech.audio.utils import logger

epochs = 20
steps_per_epoch = len(train_loader)
log_freq = 10
eval_freq = 10

for epoch in range(1, epochs + 1):
    model.train()

    avg_loss = 0
    num_corrects = 0
    num_samples = 0
    for batch_idx, batch in enumerate(train_loader):
        waveforms, labels = batch
        feats = feature_extractor(waveforms)
        feats = paddle.transpose(feats, [0, 2, 1])  # [B, N, T] -> [B, T, N]
        logits = model(feats)

        loss = criterion(logits, labels)
        loss.backward()
        optimizer.step()
        if isinstance(optimizer._learning_rate,
                      paddle.optimizer.lr.LRScheduler):
            optimizer._learning_rate.step()
        optimizer.clear_grad()

        # Calculate loss
        avg_loss += loss.numpy()[0]

        # Calculate metrics
        preds = paddle.argmax(logits, axis=1)
        num_corrects += (preds == labels).numpy().sum()
        num_samples += feats.shape[0]

        if (batch_idx + 1) % log_freq == 0:
            lr = optimizer.get_lr()
            avg_loss /= log_freq
            avg_acc = num_corrects / num_samples

            print_msg = 'Epoch={}/{}, Step={}/{}'.format(
                epoch, epochs, batch_idx + 1, steps_per_epoch)
            print_msg += ' loss={:.4f}'.format(avg_loss)
            print_msg += ' acc={:.4f}'.format(avg_acc)
            print_msg += ' lr={:.6f}'.format(lr)
            logger.train(print_msg)

            avg_loss = 0
            num_corrects = 0
            num_samples = 0

    if epoch % eval_freq == 0 and batch_idx + 1 == steps_per_epoch:
        model.eval()
        num_corrects = 0
        num_samples = 0
        with logger.processing('Evaluation on validation dataset'):
            for batch_idx, batch in enumerate(dev_loader):
                waveforms, labels = batch
                feats = feature_extractor(waveforms)
                feats = paddle.transpose(feats, [0, 2, 1])
                
                logits = model(feats)

                preds = paddle.argmax(logits, axis=1)
                num_corrects += (preds == labels).numpy().sum()
                num_samples += feats.shape[0]

        print_msg = '[Evaluation result]'
        print_msg += ' dev_acc={:.4f}'.format(num_corrects / num_samples)

        logger.eval(print_msg)

[2022-08-30 11:22:36,427] [   TRAIN] - PaddleAudio | Epoch=17/20, Step=30/35 loss=0.0292 acc=0.9938 lr=0.000100
[2022-08-30 11:22:56,232] [   TRAIN] - PaddleAudio | Epoch=18/20, Step=10/35 loss=0.1053 acc=0.9625 lr=0.000100
[2022-08-30 11:23:09,429] [   TRAIN] - PaddleAudio | Epoch=18/20, Step=20/35 loss=0.0349 acc=1.0000 lr=0.000100
[2022-08-30 11:23:22,678] [   TRAIN] - PaddleAudio | Epoch=18/20, Step=30/35 loss=0.0217 acc=1.0000 lr=0.000100
[2022-08-30 11:23:42,471] [   TRAIN] - PaddleAudio | Epoch=19/20, Step=10/35 loss=0.0464 acc=0.9875 lr=0.000100
[2022-08-30 11:23:55,696] [   TRAIN] - PaddleAudio | Epoch=19/20, Step=20/35 loss=0.0748 acc=0.9750 lr=0.000100
[2022-08-30 11:24:08,908] [   TRAIN] - PaddleAudio | Epoch=19/20, Step=30/35 loss=0.0855 acc=0.9750 lr=0.000100
[2022-08-30 11:24:28,751] [   TRAIN] - PaddleAudio | Epoch=20/20, Step=10/35 loss=0.0456 acc=0.9875 lr=0.000100
[2022-08-30 11:24:41,975] [   TRAIN] - PaddleAudio | Epoch=20/20, Step=20/35 loss=0.0383 acc=0.9875 lr=0.000100
[2022-08-30 11:24:55,153] [   TRAIN] - PaddleAudio | Epoch=20/20, Step=30/35 loss=0.0494 acc=1.0000 lr=0.000100
[2022-08-30 11:25:03,517]  Evaluation on validation dataset: - - PaddleAudio | Evaluation on validation dataset: \ - PaddleAudio | [Evaluation result] dev_acc=0.8983

四、模型预测

import glob

test_files=glob.glob("dataset/test/*.wav")
print(len(test_files))

199

top_k = 3

n_fft = 1024
win_length = 1024
hop_length = 320
f_min=50.0
f_max=16000.0

wav_file = 'dataset/test/001.wav'
waveform, sr = load(wav_file, sr=sr)
feature_extractor = LogMelSpectrogram(
    sr=sr, 
    n_fft=n_fft, 
    hop_length=hop_length, 
    win_length=win_length, 
    window='hann', 
    f_min=f_min, 
    f_max=f_max, 
    n_mels=64)
feats = feature_extractor(paddle.to_tensor(paddle.to_tensor(waveform).unsqueeze(0)))
feats = paddle.transpose(feats, [0, 2, 1])  # [B, N, T] -> [B, T, N]

logits = model(feats)
probs = nn.functional.softmax(logits, axis=1).numpy()

sorted_indices = probs[0].argsort()

print(sorted_indices)

[4 3 1 2 0 5]

6_小孩哭泣: 0.92871

1_看电视: 0.04996

3_炒菜: 0.02015

from paddlespeech.audio import load

f=open("result.csv",'w')
f.write('id,label\n')
for wav_file in test_files:
    waveform, sr = load(wav_file)
    feature_extractor = LogMelSpectrogram(
        sr=sr, 
        n_fft=n_fft, 
        hop_length=hop_length, 
        win_length=win_length, 
        window='hann', 
        f_min=f_min, 
        f_max=f_max, 
        n_mels=64)
    feats = feature_extractor(paddle.to_tensor(paddle.to_tensor(waveform).unsqueeze(0)))
    feats = paddle.transpose(feats, [0, 2, 1])  # [B, N, T] -> [B, T, N]

    logits = model(feats)
    probs = nn.functional.softmax(logits, axis=1).numpy()

    sorted_indices = probs[0].argsort()
    filename=os.path.basename(wav_file)
    label=sorted_indices[-1]+1
    # print(f'{filename}, {label}')
    f.write(f'{filename},{label}\n')      
f.close()    

下载 result.csv 提交即可得到分数。