Syllable clustering analysis-based passive acoustic monitoring technology and its application in bird monitoring

doi:10.17520/biods.2022370

Abstract

Abstract:

Aims: Passive acoustic monitoring has proven to be an effective method for monitoring bird biodiversity, as it allows for the analysis of important information such as bird songs and calls. The complexity and variations of bird songs and calls make it difficult to quickly and accurately identify bird species using voiceprint analysis. Solving this problem is essential for the successful implementation of a voiceprint-based bird diversity monitoring scheme.

Methods: This paper proposes a syllable clustering analysis-based approach for bird song/call monitoring framework. The first step is to extract syllables from voiceprint data using audio features such as pitch and frequency flatness. These syllables are then trained using a combination of unsupervised representation learning and a Dirichlet process hybrid model. The final steps are clustering the syllables and inferring their categories.

Results: (1) The analysis results show that, the proposed framework can achieve nearly 90% clustering accuracy when handling the published recordings of Lonchura striata song repository; (2) On the basis, the paper conducts unsupervised syllable clustering analysis on ten species of birds monitored in Baiyun Mountain Forest Park, Guangzhou, between April and May 2022. It verifies that the proposed framework can not only support bird species identification, but also meet the rapid species identification application requirements. This can be extended further to obtain the statistics and changes in time, frequency and quantity of various bird songs/calls.

Conclusion: The analysis results of this paper show us that, the syllable clustering-based bird song/call monitoring framework can significantly reduce the requirements for manually annotated training data. This also overcomes the shortcomings of the traditional framework in dealing with overlapping bird songs. Therefore, it provides a comprehensive solution for applications such as rapid species recognition, syllable sequence analysis, and population abundance analysis in bird diversity monitoring.

Key words: passive acoustic monitoring, birds syllable, unsupervised clustering, species identification, syllabic sequence analysis, abundance analysis

Keyi Wu, Wenda Ruan, Difeng Zhou, Qingchen Chen, Chengyun Zhang, Xinyuan Pan, Shang Yu, Yang Liu, Rongbo Xiao. Syllable clustering analysis-based passive acoustic monitoring technology and its application in bird monitoring[J]. Biodiv Sci, 2023, 31(1): 22370.

Add to citation manager EndNote|Ris|BibTeX

URL: https://www.biodiversity-science.net/EN/10.17520/biods.2022370

https://www.biodiversity-science.net/EN/Y2023/V31/I1/22370

Figures/Tables 9

Fig. 1 The whole structure of deep unsupervised syllable clustering. The sound is collected by the pickup terminal, each bird song is located and extracted by the syllable extraction algorithm, and the syllable classifier is obtained by alternate learning of representation learning and Dirichlet process mixed model (DPMM).

Table 1 The multi-feature based syllable detection and extraction algorithm proposed in this paper

1: 输入input: 包含N个样本的原始音频X_1:_N, 每帧信号样本数T, 相邻语音帧偏移参数H, 音高检测阈值θ_pitch
2: $L F = (N - T) / H + 1$ %语音帧数计算
3: $F k = 1 : L F = s p l i t (X 1 : N, T, H)$ %将原始音频X_1:N拆分成 $F k = 1 : L F = s p l i t (X 1 : N, T, H)$ 段语音帧
4: for $i ∈ 0, L F M$ do %将相邻的M个语音帧合成1个语音块
5: $s t a r t i = i × M; e n d i = M × (i + 1)$ %确定第i个语音块边界
6: $B i = {F k \| k ∈ s t a r t i, e n d i}$ %获得第i个语音块
7: $I S F B i ← < 0.1 F B i ← F l a t (B i)$ %计算第i个语音块的频率平坦度F_Bi, 根据平坦度阈值0.1, 确定音节指示向量 $I S F B i ← < 0.1 F B i ← F l a t (B i)$
8: $E B i ← power F B i ← S T F T (B i)$ %计算第i个语音块的声谱图E_Bi
9: $? I S P B i ← q p i t c h P B i ← Y I N F 0 m i n ← a r g m a x E B i$ %根据第i个语音块声谱图E_Bi, 获得最大功率对应的频点为基频 $? I S P B i ← q p i t c h P B i ← Y I N F 0 m i n ← a r g m a x E B i$ , 以此为依据进行音高检测确定当前语音块音高 $? I S P B i ← q p i t c h P B i ← Y I N F 0 m i n ← a r g m a x E B i$ , 根据音高检测阈值, 确定音高指示向量 $? I S P B i ← q p i t c h P B i ← Y I N F 0 m i n ← a r g m a x E B i$
10: $I S Z B i ← ≤ m e a n (Z C R (B i)) Z C R (B i)$ %计算第i个语音块的ZCR(B_i), 根据超过平均ZCR值, 确定ZCR指示向量 $I S Z B i ← ≤ m e a n (Z C R (B i)) Z C R (B i)$
11: $I S E B i ← max k ∈ B i (E k) > m e a n (E B i) E B i$ %根据第i个语音块的声谱图E_Bi, 根据超过当前语音块平均能量, 确定能量指示向量 $I S E B i ← max k ∈ B i (E k) > m e a n (E B i) E B i$
12:end for
13: $M a s k 1 ∈ ∪ i = 0 L F M I S P B i ∩ ∪ i = 0 L F M I S F B i$ %整合全部语音块的音高指示向量和音节指示向量, 获得融合平坦度和音高检测的音节检测掩模 $M a s k 1 ∈ ∪ i = 0 L F M I S P B i ∩ ∪ i = 0 L F M I S F B i$
14: $M a s k 2 ∈ ∪ i = 0 L F M I S Z B i ∩ ~ M a s k 1 ∩ ∪ i = 0 L F M I S E B i$ %整合全部语音块的ZCR指示向量和能量指示向量, 从平坦度和音高检测的音节检测掩模 $M a s k 2 ∈ ∪ i = 0 L F M I S Z B i ∩ ~ M a s k 1 ∩ ∪ i = 0 L F M I S E B i$ 的语音帧中, 基于能量原则和ZCR原则的音节检测掩模 $M a s k 2 ∈ ∪ i = 0 L F M I S Z B i ∩ ~ M a s k 1 ∩ ∪ i = 0 L F M I S E B i$
15: Mask $= M a s k 1 ∪ M a s k 2$ %整合平坦度、音高、能量和ZCR原则的总的音节检测掩模Mask
16: return $S y l l a b l e ? = ? X ? M a s k$ %根据音节检测掩模Mask提取音节 $S y l l a b l e ? = ? X ? M a s k$

Table 1 The multi-feature based syllable detection and extraction algorithm proposed in this paper

1: 输入input: 包含N个样本的原始音频X_1:_N, 每帧信号样本数T, 相邻语音帧偏移参数H, 音高检测阈值θ_pitch
2: $L F = (N - T) / H + 1$ %语音帧数计算
3: $F k = 1 : L F = s p l i t (X 1 : N, T, H)$ %将原始音频X_1:N拆分成 $F k = 1 : L F = s p l i t (X 1 : N, T, H)$ 段语音帧
4: for $i ∈ 0, L F M$ do %将相邻的M个语音帧合成1个语音块
5: $s t a r t i = i × M; e n d i = M × (i + 1)$ %确定第i个语音块边界
6: $B i = {F k \| k ∈ s t a r t i, e n d i}$ %获得第i个语音块
7: $I S F B i ← < 0.1 F B i ← F l a t (B i)$ %计算第i个语音块的频率平坦度F_Bi, 根据平坦度阈值0.1, 确定音节指示向量 $I S F B i ← < 0.1 F B i ← F l a t (B i)$
8: $E B i ← power F B i ← S T F T (B i)$ %计算第i个语音块的声谱图E_Bi
9: $? I S P B i ← q p i t c h P B i ← Y I N F 0 m i n ← a r g m a x E B i$ %根据第i个语音块声谱图E_Bi, 获得最大功率对应的频点为基频 $? I S P B i ← q p i t c h P B i ← Y I N F 0 m i n ← a r g m a x E B i$ , 以此为依据进行音高检测确定当前语音块音高 $? I S P B i ← q p i t c h P B i ← Y I N F 0 m i n ← a r g m a x E B i$ , 根据音高检测阈值, 确定音高指示向量 $? I S P B i ← q p i t c h P B i ← Y I N F 0 m i n ← a r g m a x E B i$
10: $I S Z B i ← ≤ m e a n (Z C R (B i)) Z C R (B i)$ %计算第i个语音块的ZCR(B_i), 根据超过平均ZCR值, 确定ZCR指示向量 $I S Z B i ← ≤ m e a n (Z C R (B i)) Z C R (B i)$
11: $I S E B i ← max k ∈ B i (E k) > m e a n (E B i) E B i$ %根据第i个语音块的声谱图E_Bi, 根据超过当前语音块平均能量, 确定能量指示向量 $I S E B i ← max k ∈ B i (E k) > m e a n (E B i) E B i$
12:end for
13: $M a s k 1 ∈ ∪ i = 0 L F M I S P B i ∩ ∪ i = 0 L F M I S F B i$ %整合全部语音块的音高指示向量和音节指示向量, 获得融合平坦度和音高检测的音节检测掩模 $M a s k 1 ∈ ∪ i = 0 L F M I S P B i ∩ ∪ i = 0 L F M I S F B i$
14: $M a s k 2 ∈ ∪ i = 0 L F M I S Z B i ∩ ~ M a s k 1 ∩ ∪ i = 0 L F M I S E B i$ %整合全部语音块的ZCR指示向量和能量指示向量, 从平坦度和音高检测的音节检测掩模 $M a s k 2 ∈ ∪ i = 0 L F M I S Z B i ∩ ~ M a s k 1 ∩ ∪ i = 0 L F M I S E B i$ 的语音帧中, 基于能量原则和ZCR原则的音节检测掩模 $M a s k 2 ∈ ∪ i = 0 L F M I S Z B i ∩ ~ M a s k 1 ∩ ∪ i = 0 L F M I S E B i$
15: Mask $= M a s k 1 ∪ M a s k 2$ %整合平坦度、音高、能量和ZCR原则的总的音节检测掩模Mask
16: return $S y l l a b l e ? = ? X ? M a s k$ %根据音节检测掩模Mask提取音节 $S y l l a b l e ? = ? X ? M a s k$

Fig. 2 The syllables of Lonchura striata (top) and Baiyun Mountain birds (bottom) detected and automatically labeled. The part highlighted with white box in the spectrogram are syllables.

Fig. 3 Schematic diagram of bird song syllable clustering based on variational encoder. The encoder projects the syllables into a low-dimensional latent space, and the decoder reconstructs the syllables using the mean and variance vectors.

Fig. 4 Syllable detection and annotation results after unsupervised clustering of Lonchura striata repertoire (Bird 032312). The white boxes are marked as syllable ranges, the number tag above the syllable is a pseudo tag assigned by unsupervised clustering, and the number below the syllable corresponds to the syllable tag marked by the expert.

Fig. 5 The syllable clustering results of Lonchura striata repertoire Bird 032312 are visualized; the 64-dimensional features are reduced to 3 and 2 dimensions using the t-SNE dimensionality reduction method.

Fig. 6 Comparison of the clustering performance of traditional clustering algorithms and the proposed deep learning-based methods in the Lonchura striata song library. The abscissa is the number of each bird, the ordinate (left) is the clustering accuracy, and the ordinate (right) represents the number of syllables. The red line is the number of syllable types, and the cluster number of each bird is sorted in ascending order according to the number of syllable species of each bird.

Fig. 7 Statistics on the number of syllables of birds at a monitoring site in Baiyunshan Park: (left) the number of syllables and (right) the number of syllables. The number and types of bird syllables are count every 6 days from April 1, 2022 to May 9, 2022. The statistical time period is from 5:00 to 9:00 in the morning, and from 16:00 to 21:00 in the evening.

Fig. 8 Syllable counting based quantity statistics of ten bird species in Baiyun Mountain Park

References 40

[1]	Alghamdi A, Mehtab T, Iqbal R, Leeza M, Islam N, Hamdi M, Shaikh A (2021) Automatic classification of monosyllabic and multisyllabic birds using PDHF. Electronics, 10, 624. DOI URL
[2]	Baker MC, Cunningham MA (1985) The biology of bird-song dialects. Behavioral and Brain Sciences, 8, 85-100. DOI URL
[3]	Bilger HT, Vertosick E, Vickers A, Kaczmarek K, Prum RO (2021) Higher-order musical temporal structure in bird song. Frontiers in Psychology, 12, 629456.
[4]	Botero CA, Rossman RJ, Caro LM, Stenzler LM, Lovette IJ, de Kort SR, Vehrencamp SL (2009) Syllable type consistency is related to age, social status and reproductive success in the tropical mockingbird. Animal Behaviour, 77, 701-706. DOI URL
[5]	Bradfer-Lawrence T, Gardner N, Bunnefeld L, Bunnefeld N, Willis SG, Dent DH (2019) Guidelines for the use of acoustic indices in environmental research. Methods in Ecology and Evolution, 10, 1796-1807. DOI
[6]	Cohen Y, Nicholson D, Gardner TJ (2020) TweetyNet: A neural network that enables high-throughput, automated annotation of birdsong. bioRxiv, doi:10.1101/2020.08.28. 272088. DOI
[7]	Cui P, Deng WH (2007) Review on the status and development of bird community research. Chinese Journal of Zoology, 42, 149-158. (in Chinese with English abstract)
	[崔鹏, 邓文洪 (2007) 鸟类群落研究进展. 动物学杂志, 42, 149-158.]
[8]	Fagerlund S (2004) Automatic Recognition of Bird Species by Their Sounds. PhD thesis, Helsinki University of Technology, Helsinki.
[9]	Geberzahn N, Gahr M (2011) Undirected (solitary) birdsong in female and male blue-capped cordon-bleus (Uraeginthus cyanocephalus) and its endocrine correlates. PLoS ONE, 6, e26485.
[10]	Große Ruse M, Hasselquist D, Hansson B, Tarka M, Sandsten M (2016) Automated analysis of song structure in complex birdsongs. Animal Behaviour, 112, 39-51. DOI URL
[11]	Gutiérrez RJ, Cody M, Courtney S, Franklin AB (2007) The invasion of barred owls and its potential effect on the spotted owl: A conservation conundrum. Biological Invasions, 9, 181-196. DOI URL
[12]	He KM, Zhang XY, Ren SQ, Sun J (2016) Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA. IEEE, 770-778.
[13]	Heck M, Sakti S, Nakamura S (2018) Dirichlet process mixture of mixtures model for unsupervised subword modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26, 2027-2042. DOI URL
[14]	Illes AE (2015) Context of female bias in song repertoire size, singing effort, and singing independence in a cooperatively breeding songbird. Behavioral Ecology and Sociobiology, 69, 139-150. DOI URL
[15]	Katahira K, Suzuki K, Okanoya K, Okada M (2011) Complex sequencing rules of birdsong can be explained by simple hidden Markov processes. PLoS ONE, 6, e24516.
[16]	Kingma DP, Welling M (2013) Auto-encoding variational Bayes. arXiv: 1312.6114. https://arxiv.org/abs/1312.6114.
[17]	Kong QQ, Cao Y, Iqbal T, Wang YX, Wang WW, Plumbley MD (2020) PANNs: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28, 2880-2894. DOI URL
[18]	Koumura T, Okanoya K (2016) Automatic recognition of element classes and boundaries in the birdsong with variable sequences. PLoS ONE, 11, e0159188.
[19]	Lindsay B, McLachlan GL, Basford KE, Dekker M (1989) Mixture models: Inference and applications to clustering. Journal of the American Statistical Association, 84, 337. DOI URL
[20]	Lobato M, Vellema M, Gahr C, Leitão A, de Lima SMA, Geberzahn N, Gahr M (2015) Mismatch in sexual dimorphism of developing song and song control system in blue-capped cordon-bleus, a songbird species with singing females and males. Frontiers in Ecology and Evolution, 3, 117.
[21]	Marck A, Vortman Y, Kolodny O, Lavner Y (2022) Identification, analysis and characterization of base units of bird vocal communication: The white spectacled bulbul (Pycnonotus xanthopygos) as a case study. Frontiers in Behavioral Neuroscience, 15, 812939.
[22]	Micheli-Tzanakou E (2017) Supervised and Unsupervised Pattern Recognition:Feature Extraction and Computational intelligence. CRC Press, Boca Raton.
[23]	Morita T, Koda H, Okanoya K, Tachibana RO (2021) Measuring context dependency in birdsong using artificial neural networks. PLoS Computational Biology, 17, e1009707.
[24]	O’Reilly C, Marples NM, Kelly DJ, Harte N (2016) YIN-Bird: Improved Pitch Tracking for Bird Vocalisations Interspeech, ISCA. doi: 10.21437/Interspeech.2016-90. DOI
[25]	Pillay R, Fletcher RJ Jr, Sieving KE, Udell BJ, Bernard H (2019) Bioacoustic monitoring reveals shifts in breeding songbird populations and singing behaviour with selective logging in tropical forests. Journal of Applied Ecology, 56, 2482-2492. DOI URL
[26]	Planqué R, Britton NF, Slabbekoorn H (2014) On the maintenance of bird song dialects. Journal of Mathematical Biology, 68, 505-531. DOI PMID
[27]	Potamitis I, Ntalampiras S, Jahn O, Riede K (2014) Automatic bird sound detection in long real-field recordings: Applications and tools. Applied Acoustics, 80, 1-9. DOI URL
[28]	Sainburg T, Thielk M, Gentner TQ (2019) Latent space visualization, characterization, and generation of diverse vocal communication signals. bioRxiv, doi: 10.1101/870311. DOI
[29]	Snell-Rood EC (2012) The effect of climate on acoustic signals: Does atmospheric sound absorption matter for bird song and bat echolocation. The Journal of the Acoustical Society of America, 131, 1650-1658. DOI URL
[30]	Steinfath E, Palacios-Muñoz A, Rottschäfer JR, Yuezak D, Clemens J (2021) Fast and accurate annotation of acoustic signals with deep neural networks. eLife, 10, 68837.
[31]	Stowell D, Wood MD, Pamuła H, Stylianou Y, Glotin H (2019) Automatic acoustic detection of birds through deep learning: The first Bird Audio Detection challenge. Methods in Ecology and Evolution, 10, 368-380. DOI
[32]	Suzuki TN (2021) Animal linguistics: Exploring referentiality and compositionality in bird calls. Ecological Research, 36, 221-231. DOI URL
[33]	Torres-García AA, Mendoza-Montoya O, Reyes-García CA, Villaseñor-Pineda L (2021) Biosignal Processing and Classification Using Computational Learning and Intelligence, pp. 3-6. Elsevier, Amsterdam.
[34]	Von Luxburg U (2007) A tutorial on spectral clustering. Statistics and Computing, 17, 395-416.
[35]	Wang Q, Lü XG (2007) Application of water bird to monitor and evaluate wetland ecosystem. Wetland Science, 5, 274-281. (in Chinese with English abstract)
	[王强, 吕宪国 (2007) 鸟类在湿地生态系统监测与评价中的应用. 湿地科学, 5, 274-281.]
[36]	Xie J, Towsey M, Truskinger A, Eichinski P, Zhang JL, Roe P (2015) Acoustic classification of Australian anurans using syllable features. 2015 IEEE Tenth International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP). April 7-9, 2015, Singapore. IEEE, 1-6.
[37]	Yang CC, Cai Y, Liang W (2009) Bird diversity in parks of Guangzhou. Journal of Hainan Normal University (Natural Science), 22, 196-199. (in Chinese with English abstract)
	[杨灿朝, 蔡燕, 梁伟 (2009) 广州市区各公园的鸟类多样性比较. 海南师范大学学报(自然科学版), 22, 196-199.]
[38]	Yang LX, Cheung NM, Li JY, Fang J (2019) Deep clustering by Gaussian mixture variational autoencoders with graph embedding. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). October 27-November 2, 2019, Seoul, Korea (South). IEEE, 6439-6448.
[39]	Zhang T, Ramakrishnan R, Livny M (1996) Birch: An efficient data clustering method for very large databases. SIGMOD Record (ACM Special Interest Group on Management of Data), 25, 103-114.
[40]	Zhou S, Xu HJ, Zheng ZN, Chen JW li Z, Bu JJ, Wu J, Wang X, Zhu WW, Ester M (2022) A comprehensive survey on deep clustering: Taxonomy, challenges, and future directions. arXiv: 2206.07579. https://arxiv.org/abs/2206.07579.