PLoS ONE

Clubb, Jo;Towlson, Chris;Barrett, Steve

doi: 10.1371/journal.pone.0262274pmid: 35061784

Introduction Training games are used extensively in soccer training with a variety of formats, with the aim to develop players’ technical ability and tactical awareness, while concurrently targeting physiological capacities [1–3]. The inclusion and organisation of such training is considered to provide an internal training response (physiological [4,5], biomechanical [6] and psychological [4]), which practitioners attempt to capture via measures of internal load, such as heart rate or ratings of perceived exertion [4]. This response is largely governed by the accrued external training load (e.g., distances covered at different speeds [5]), which is defined as the external stimulus applied to the athlete measured independently of their internal characteristics [4]. Modifications to the constraints applied to training games have been shown to substantially influence these training processes and outcomes [1,7]. As such, practitioners must have an understanding of how these constraints may be manipulated to target technical, perceptual and physical outcomes, relative to player development and match-play performance [3]. In soccer these constraints can include the number of players, pitch size, training prescription (sets, repetitions and work: rest ratios), technical rules (limited touches, position on pitch to score), the inclusion of goalkeepers and coach encouragement [1,8,9]. Yet while such training games may aim to develop physical capacities due to the stochastic nature of this training, it may be difficult to precisely prescribe and periodise the external training load undertaken [10]. This is important, as this method of training is often employed in situations where training loads need to be well-considered (i.e., to elicit certain physiological adaptations, return to play etc.). Despite this, research has suggested that monitoring the internal and/or external training loads undertaken by players during training drills may allow for fatigue detection [11,12]. Specifically, regularly performing “standardised” versions of training games, whereby such drill constraints are kept consistent, may be used to assess fatigue status in-situ [11]. However, Rowell and colleagues [11] presented unclear findings suggesting individual variability was demonstrated in some external load metrics. Further understanding of the measurement properties of such metrics within standardised training games is required, in order to determine the potential noise that might be associated with such formats, especially if they are to be used as an assessment of fatigue status. It is reasonable to assume that variability in training loads (within- and between-individuals) during training games may extend past constraint alternations, having implications for physical training and fatigue monitoring strategies. Yet while research has investigated some of these measurement properties, it has primarily been conducted with focus on participant numbers and specifically, smaller small sided games (SSG) (e.g. 4v4, 2v2 [1,2]), with no research to date investigating the reliability and sensitivity of commonly used medium and large sided games in soccer. Soccer practitioners commonly monitor external training loads using micro-electromechanical systems (MEMS), such as global positioning systems (GPS) and accelerometers [13,14]. Here, metrics such as total distance covered, number of efforts above absolute and/or relative speed thresholds, maximum velocity, and acceleration/deceleration efforts across different formats are typically reported for analysis [7,15]. Caution has been advised when using MEMS devices to measure external training load during training drills due to some difficulty in capturing complex movement patterns (coupled with decreased task duration [16,17]). However, if practitioners appropriately consider the validity and reliability of devices used and metrics analysed these may be overcome [17,18]. In addition, traditional locomotor analysis fails to account for movements in multiple planes of movement such as tackles, changing direction and jumping [19,20]. Tri-axial accelerometers have been increasingly used as a measure of external training load in team sports, most commonly utilising PlayerLoad™ (PL) [20,21]. High levels of validity and reliability have been shown for overall PL and each of the individual planes (vertical (PLV), medio-lateral (PLML) and anterior-posterior (PLAP)) between soccer matches (coefficient of variation; CV = 6.4%) and within a soccer specific simulation (Intraclass coefficient correlation; ICC = 0.80–0.99) [20]. Consequently, recommendations to use PL as a measure of external training load would allow practitioners to detect meaningful differences in accumulated load [20,21]. Despite this, limited information is available on the variability of PL during training games and the effect of different formats [7,15]. This information will likely be useful to practitioners given that it provides greater detail on the movement strategies performed by players during such games. Further, despite a body of research examining SSG [3,8,9], insufficient attention has been paid to the variability of movement patterns during such training drills [22]. Yet, external training load is often central to training prescription and return to play strategies [23,24] as well as to player support monitoring systems [25]. As such, a greater understanding of the measurement properties of this training load (dose) is required. Therefore, the aim of the current study was to examine the measurement properties of external training load measures, specifically the within- and between-subject reliability and sensitivity, with different formats of standardised training games commonly performed in soccer. Materials & methods Subjects Eighty-eight male, elite soccer players (Age: 26.5 ± 5.8 years; stature: 1.82 ± 0.07m; body-mass: 78.8 ± 7.7kg) were recruited from two English professional teams, one in the top domestic tier (Premier League) and the other in the second tier (Championship). The study gained ethical approval study (1011137) from a university departmental ethics committee prior to the commencement of the study. As the data reported in this retrospective study was collected as part of the routine data monitoring of players in industry practice, informed consent was not deemed necessary [26]. Procedures Data was collected during the 2014/2015, 2015/2016 and 2016/2017 seasons from two English professional soccer teams. To provide valid and reliable information, each outfield player wore a MEMS device (Optimeye S5, Catapult Sports, Melbourne, Australia; Firmware version- 6.88–6.72), in a customized, tight-fitting neoprene garment (positioned between the scapulae) [27], as part of their daily monitoring routines within their respective training sessions at each club. These devices were taken outside and activated 15–30 minutes beforehand to attenuate erroneous data owing to poor GPS signal quality [28]. Each player wore the same unit for each session. Accelerometry. The Optimeye S5 MEMS device contains a tri-axial piezoelectric linear accelerometer (Kionix: KXP94) sampling at a frequency of 100-Hz. The output of the accelerometer measures ±13g, with each device containing its own microprocessor with a 1GB flash memory and USB interface to store and download data. From this, PL was calculated by and exported from the manufacturer’s software as the sum of the instantaneous rate of change from the individual planes (PLV, PLML, and PLAP) [20,21], expressed in arbitrary units (au). The percentage contribution of the individual component planes to PL were also exported from the software. As per previous studies [11,20,29], PL was presented relative to the duration of the game (PL/min) and integrated with GPS data to calculate PL/metre. Data were recorded throughout training drills using the Catapult software (Sprint 5.1.7, Catapult Sports, Melbourne, Australia). Prior to the start of each season, units were calibrated using the manufacturers jig to ensure values were set within the manufacturers guidelines [28]. Specifically, the device was orientated and placed stationary in each plane of movement and recordings were set at 1g for that position to reduce any bias or drift [20]. Monthly checks of the calibration values were monitored to ensure the calibration values remained within the manufacturer’s calibration values throughout the testing period. Time-motion analysis. The Optimeye S5 contains a 10-Hz GPS chip to record time-motion data. External load variables monitored included total distance (TD), metres per minute (m/min) and high-speed running (HSR). Commonly, HSR has been assessed via absolute and/or relative thresholds [14,30] therefore; both were included in this study. At one team, an absolute threshold (HSRa) of 5.5m/s for HSR was used [31]. At the other team, a relative threshold (HSRr) of >65% of each individual’s maximal velocity for HSR was used [32]. Each individual’s maximum velocity was determined by 10-Hz GPS data tracked across the season, as previous work has shown no significant differences exist for speed measures captured using timing gates and GPS technology [33]. Peak velocities reached by individuals were monitored daily and when a new maximum was reached, the individual’s maximum velocity was changed on the tracking system from that point onwards. For the purpose of this study, all data was then updated retrospectively with the players’ maximum velocity achieved throughout the season. As per previous methods [34], all dwell times for the variables were set to 0.2s. Data was only included if the number of satellites exceeded 6, a horizontal displacement of positioning (HDOP) was less than 1.5 and the IMF (intelligent motion filter) was switched on in the software. Standardised training games. The training drills were prescribed by the respective head coaches with no intervention by sports science staff. Three different formats according to the number of players involved, standardised for all other constraints as shown in Table 1, were included in the study. At one team, an 11v11 (trials = 14; cases = 236) and 10v10 (trials = 10; cases = 432) format were performed. At the other team, a 7v7+6 (trials = 6; cases = 92) was performed. The games were consistently played on the training day prior to the next match and at least 48 hours after the previous match. Coaches were asked to maintain a consistent level of encouragement throughout, with trials excluded if any alterations were made to the games. Subjects were included if they had carried out at least three trials of the same game format. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Standardised conditions of game formats. https://doi.org/10.1371/journal.pone.0262274.t001 Statistical analyses Data are presented as mean ± standard deviation (SD). Between-trial reliability of external training load variables for each game format was assessed using the percentage of coefficient of variation (CV%). This was calculated for each external load variable within each game format, using a custom spreadsheet in Microsoft Excel [35]. In order to assess the variability across trials for each player, within-subject CV% was also calculated for each external load variable across the standardised game formats. This was calculated for each individual using their between-day variation by dividing the individual’s SD by the individual’s mean and multiplying by 100. In order to demonstrate the differences and applications between commonly utilised reliability measurement properties, the authors present findings using both—between-trial and within-subject—methods. The smallest worthwhile change (SWC) can be used to assess meaningful differences in performance [36]. The SWC was calculated as 0.2 of the between-player SD. In addition, test-retest reliability of the external training load variables were reported as the intra-class correlation coefficient (ICC) ± 90% confidence intervals (CI) using a custom spreadsheet [35]. The following criteria were used to interpret the ICC coefficients: < 0.50 poor, 0.50–0.75 moderate, 0.75–0.90 good, ≥ 0.90 excellent [37]. It is also important to consider the sensitivity of a measure, because absolute reliability does not necessarily mean a metric is sensitive to detecting a change (signal) greater than the error (noise) [38]. The signal-to-noise ratio (SNR) was calculated for each external load metric by dividing the weekly variation in a measure (% change in group mean) by the between-trial reliability (CV%) and then taking the average of all trials [38]. This was assessed using the same reliability spreadsheet [35]. The SNR was classified as good if greater than 1 and poor if less than 1 [38]. In response to calls in sports science to utilise data analysis and visualisation techniques that allow the reader to appreciate distribution and outliers, a violin plot was used to present within-subject CV% [39]. A violin plot is a combination of a box plot and a density plot. The box plot represents the median, the interquartile range and 95% confidence limits. The density plot represents the distribution shape, with a wider plot representing a higher frequency. The shape of the violin plot represents the probability density, with a higher likelihood of seeing an individual data point fall within the thicker part of the plot [39]. Outliers are shown as individual data points. By using this method to display the within-subject CV%, the reader can observe the variability in reliability of each external load variable within their playing group. Subjects Eighty-eight male, elite soccer players (Age: 26.5 ± 5.8 years; stature: 1.82 ± 0.07m; body-mass: 78.8 ± 7.7kg) were recruited from two English professional teams, one in the top domestic tier (Premier League) and the other in the second tier (Championship). The study gained ethical approval study (1011137) from a university departmental ethics committee prior to the commencement of the study. As the data reported in this retrospective study was collected as part of the routine data monitoring of players in industry practice, informed consent was not deemed necessary [26]. Procedures Data was collected during the 2014/2015, 2015/2016 and 2016/2017 seasons from two English professional soccer teams. To provide valid and reliable information, each outfield player wore a MEMS device (Optimeye S5, Catapult Sports, Melbourne, Australia; Firmware version- 6.88–6.72), in a customized, tight-fitting neoprene garment (positioned between the scapulae) [27], as part of their daily monitoring routines within their respective training sessions at each club. These devices were taken outside and activated 15–30 minutes beforehand to attenuate erroneous data owing to poor GPS signal quality [28]. Each player wore the same unit for each session. Accelerometry. The Optimeye S5 MEMS device contains a tri-axial piezoelectric linear accelerometer (Kionix: KXP94) sampling at a frequency of 100-Hz. The output of the accelerometer measures ±13g, with each device containing its own microprocessor with a 1GB flash memory and USB interface to store and download data. From this, PL was calculated by and exported from the manufacturer’s software as the sum of the instantaneous rate of change from the individual planes (PLV, PLML, and PLAP) [20,21], expressed in arbitrary units (au). The percentage contribution of the individual component planes to PL were also exported from the software. As per previous studies [11,20,29], PL was presented relative to the duration of the game (PL/min) and integrated with GPS data to calculate PL/metre. Data were recorded throughout training drills using the Catapult software (Sprint 5.1.7, Catapult Sports, Melbourne, Australia). Prior to the start of each season, units were calibrated using the manufacturers jig to ensure values were set within the manufacturers guidelines [28]. Specifically, the device was orientated and placed stationary in each plane of movement and recordings were set at 1g for that position to reduce any bias or drift [20]. Monthly checks of the calibration values were monitored to ensure the calibration values remained within the manufacturer’s calibration values throughout the testing period. Time-motion analysis. The Optimeye S5 contains a 10-Hz GPS chip to record time-motion data. External load variables monitored included total distance (TD), metres per minute (m/min) and high-speed running (HSR). Commonly, HSR has been assessed via absolute and/or relative thresholds [14,30] therefore; both were included in this study. At one team, an absolute threshold (HSRa) of 5.5m/s for HSR was used [31]. At the other team, a relative threshold (HSRr) of >65% of each individual’s maximal velocity for HSR was used [32]. Each individual’s maximum velocity was determined by 10-Hz GPS data tracked across the season, as previous work has shown no significant differences exist for speed measures captured using timing gates and GPS technology [33]. Peak velocities reached by individuals were monitored daily and when a new maximum was reached, the individual’s maximum velocity was changed on the tracking system from that point onwards. For the purpose of this study, all data was then updated retrospectively with the players’ maximum velocity achieved throughout the season. As per previous methods [34], all dwell times for the variables were set to 0.2s. Data was only included if the number of satellites exceeded 6, a horizontal displacement of positioning (HDOP) was less than 1.5 and the IMF (intelligent motion filter) was switched on in the software. Standardised training games. The training drills were prescribed by the respective head coaches with no intervention by sports science staff. Three different formats according to the number of players involved, standardised for all other constraints as shown in Table 1, were included in the study. At one team, an 11v11 (trials = 14; cases = 236) and 10v10 (trials = 10; cases = 432) format were performed. At the other team, a 7v7+6 (trials = 6; cases = 92) was performed. The games were consistently played on the training day prior to the next match and at least 48 hours after the previous match. Coaches were asked to maintain a consistent level of encouragement throughout, with trials excluded if any alterations were made to the games. Subjects were included if they had carried out at least three trials of the same game format. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Standardised conditions of game formats. https://doi.org/10.1371/journal.pone.0262274.t001 Accelerometry. The Optimeye S5 MEMS device contains a tri-axial piezoelectric linear accelerometer (Kionix: KXP94) sampling at a frequency of 100-Hz. The output of the accelerometer measures ±13g, with each device containing its own microprocessor with a 1GB flash memory and USB interface to store and download data. From this, PL was calculated by and exported from the manufacturer’s software as the sum of the instantaneous rate of change from the individual planes (PLV, PLML, and PLAP) [20,21], expressed in arbitrary units (au). The percentage contribution of the individual component planes to PL were also exported from the software. As per previous studies [11,20,29], PL was presented relative to the duration of the game (PL/min) and integrated with GPS data to calculate PL/metre. Data were recorded throughout training drills using the Catapult software (Sprint 5.1.7, Catapult Sports, Melbourne, Australia). Prior to the start of each season, units were calibrated using the manufacturers jig to ensure values were set within the manufacturers guidelines [28]. Specifically, the device was orientated and placed stationary in each plane of movement and recordings were set at 1g for that position to reduce any bias or drift [20]. Monthly checks of the calibration values were monitored to ensure the calibration values remained within the manufacturer’s calibration values throughout the testing period. Time-motion analysis. The Optimeye S5 contains a 10-Hz GPS chip to record time-motion data. External load variables monitored included total distance (TD), metres per minute (m/min) and high-speed running (HSR). Commonly, HSR has been assessed via absolute and/or relative thresholds [14,30] therefore; both were included in this study. At one team, an absolute threshold (HSRa) of 5.5m/s for HSR was used [31]. At the other team, a relative threshold (HSRr) of >65% of each individual’s maximal velocity for HSR was used [32]. Each individual’s maximum velocity was determined by 10-Hz GPS data tracked across the season, as previous work has shown no significant differences exist for speed measures captured using timing gates and GPS technology [33]. Peak velocities reached by individuals were monitored daily and when a new maximum was reached, the individual’s maximum velocity was changed on the tracking system from that point onwards. For the purpose of this study, all data was then updated retrospectively with the players’ maximum velocity achieved throughout the season. As per previous methods [34], all dwell times for the variables were set to 0.2s. Data was only included if the number of satellites exceeded 6, a horizontal displacement of positioning (HDOP) was less than 1.5 and the IMF (intelligent motion filter) was switched on in the software. Standardised training games. The training drills were prescribed by the respective head coaches with no intervention by sports science staff. Three different formats according to the number of players involved, standardised for all other constraints as shown in Table 1, were included in the study. At one team, an 11v11 (trials = 14; cases = 236) and 10v10 (trials = 10; cases = 432) format were performed. At the other team, a 7v7+6 (trials = 6; cases = 92) was performed. The games were consistently played on the training day prior to the next match and at least 48 hours after the previous match. Coaches were asked to maintain a consistent level of encouragement throughout, with trials excluded if any alterations were made to the games. Subjects were included if they had carried out at least three trials of the same game format. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Standardised conditions of game formats. https://doi.org/10.1371/journal.pone.0262274.t001 Statistical analyses Data are presented as mean ± standard deviation (SD). Between-trial reliability of external training load variables for each game format was assessed using the percentage of coefficient of variation (CV%). This was calculated for each external load variable within each game format, using a custom spreadsheet in Microsoft Excel [35]. In order to assess the variability across trials for each player, within-subject CV% was also calculated for each external load variable across the standardised game formats. This was calculated for each individual using their between-day variation by dividing the individual’s SD by the individual’s mean and multiplying by 100. In order to demonstrate the differences and applications between commonly utilised reliability measurement properties, the authors present findings using both—between-trial and within-subject—methods. The smallest worthwhile change (SWC) can be used to assess meaningful differences in performance [36]. The SWC was calculated as 0.2 of the between-player SD. In addition, test-retest reliability of the external training load variables were reported as the intra-class correlation coefficient (ICC) ± 90% confidence intervals (CI) using a custom spreadsheet [35]. The following criteria were used to interpret the ICC coefficients: < 0.50 poor, 0.50–0.75 moderate, 0.75–0.90 good, ≥ 0.90 excellent [37]. It is also important to consider the sensitivity of a measure, because absolute reliability does not necessarily mean a metric is sensitive to detecting a change (signal) greater than the error (noise) [38]. The signal-to-noise ratio (SNR) was calculated for each external load metric by dividing the weekly variation in a measure (% change in group mean) by the between-trial reliability (CV%) and then taking the average of all trials [38]. This was assessed using the same reliability spreadsheet [35]. The SNR was classified as good if greater than 1 and poor if less than 1 [38]. In response to calls in sports science to utilise data analysis and visualisation techniques that allow the reader to appreciate distribution and outliers, a violin plot was used to present within-subject CV% [39]. A violin plot is a combination of a box plot and a density plot. The box plot represents the median, the interquartile range and 95% confidence limits. The density plot represents the distribution shape, with a wider plot representing a higher frequency. The shape of the violin plot represents the probability density, with a higher likelihood of seeing an individual data point fall within the thicker part of the plot [39]. Outliers are shown as individual data points. By using this method to display the within-subject CV%, the reader can observe the variability in reliability of each external load variable within their playing group. Results Mean external training load variables for each game format and the SWC are shown in Table 2. The 10v10 condition elicited the highest mean TD (2115.2 ± 243.7 m) in comparison to the 11v11 (2078.6 ± 250.5 m) and 7v7+6 (1106.7 ± 136.0 m). There was a similar pattern with the mean PL in the 10v10 (208.2 ± 37.2 AU) compared to the 11v11 (200.8 ± 38.1 AU) and 7v7+6 (113.8 ± 18.6 AU) formats. Differences in external load metrics across the different game formats were not statistically evaluated. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Activity profile metrics and reliability for different formats of standardised soccer games. https://doi.org/10.1371/journal.pone.0262274.t002 The between-subject CV, SNR, and ICC of the external load metrics are presented in Table 2. Data for TD demonstrated good (1.6 to 4.6) SNR in all three formats. Reliability assessed via the ICC was moderate-to-good for the 10v10 (0.73 to 0.83) and 11v11 (0.64 to 0.84). However, values ranged from poor-to-good (0.41 to 0.76) for the 7v7+6. That said, relative distance expressed as m/min for the 7v7+6 showed moderate-to-good (0.45 to 0.78) reliability. However, the 10v10 and 7v7+6 formats displayed poor sensitivity (0.7 to 0.8) for relative distance. Activities performed above both absolute and relative HSR thresholds demonstrated poor reliability (between-subject CV% = 51–103%; ICC = 0.03–0.53; within-subject CV% = 19–244%) across all three game formats. Given the poor reliability results, the SNR analysis for both HSR measures returned errors so this data is omitted from Table 2. The PL, PL/min and PL/metre data demonstrated between-subject CV% below 10% for all three formats. The ICC ranged from moderate-to-excellent (0.79 to 0.91). However, when PL was calculated per minute or metre, the SNR was poor (0.4 to 0.8). The percentage contribution of the individual planes to PL demonstrated the lowest between-subject CV% (2.0 to 6.7%). However, the ICC ranged from poor to good, with a lower correlation for each respective metric in the 7v7+6 compared to the 10v10 and 11v11 drills. In the 11v11 format, PLAP (%) and PLV (%) showed good sensitivity (1.2 to 1.4). Across all the external load metrics, the ICC was lower in the 7v7+6 format than 10v10 and 11v11. Within-subject CV% are displayed in the violin plot in Fig 1. Fig 1 highlights the existence of outliers that account for the large range of within-subject variation, such as PL and PL/min across all game formats. The figure also indicates the higher density of data around the mean in the percent contribution of the individual planes to PL, representing lower within-subject variation in these metrics. HSR metrics were excluded from this figure due to the poor reliability observed. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. A violin plot of the within-subject CV for each external training load measure across three different SSG formats. TD—Total distance; m/min—metres per minute; PL—PlayerLoad™; PL/min—PlayerLoad™ per minute; PL/metre—PlayerLoad™ per metre; PLAP (%)—% contribution to PlayerLoad™ in the Anterior-Posterior plane; PLML (%)—% contribution to PlayerLoad™ in the Medial-Lateral plane; PLV (%)—% contribution to PlayerLoad™ in the Vertical plane; CV%—Coefficient of variation. https://doi.org/10.1371/journal.pone.0262274.g001 Discussion Many research studies have explored the effects of different training game formats on external training load (1, 7, 9]. In order to assess when changes in external load variables across such training games are meaningful, practitioners require an understanding of the measurement properties associated with such metrics in such settings. Recently, a greater emphasis has been placed on using such drills to drive fatigue monitoring and training strategies [11,12] and therefore, understanding group and individual movement variance is critical. The current study aimed to examine the measurement properties of external training load variables across three different training game formats. The main findings of the present study are four-fold: 1) cumulative variables of TD and PL demonstrated moderate-to-good reliability and SNR, suggesting these metrics may be sensitive to track group changes across trials. 2) Conversely, m/min, PL/min and PL/metre may not be sensitive to track changes in intensity as the noise was greater than the signal (SNR<1, except for m/min in the 11v11 format). 3) Both absolute and relative HSR showed poor reliability across the three game formats. 4) The percentage contribution of individual component planes to PL demonstrated high variation, with poor-to-good reliability according to the ICC across all formats. The range of within-subject reliability across all other variables highlights the need to consider reliability on an individual athlete level in the applied environment. Understanding the reliability of a given variable will help sport scientists and practitioners to calculate a meaningful difference within that variable. Across different training game formats, variables such as TD, m/min, and HSR activities have previously been assessed for reliability [9,40]. Within the current study, TD and m/min demonstrated good between-subject CV% between 6 and 8% across all SSG formats. This is similar to previous findings that showed good reliability in total distance covered in 1v1, 2v2 (CV% = 6.1–7.9%; [9]) and 6v6 games (CV% < 5%; [40]). Therefore, given this level of reliability, practitioners can utilise changes in TD and m/min to calculate meaningful differences across trials in these formats of standardised games. In contrast, HSRa and HSRr showed poor reliability across the three formats, similar to previous research examining other game formats (2v2, 4v4, 6v6) [15]. While the reliability of HSRa in 6v6 games was enhanced (CV% = 12–17%) compared to the present study, the authors still concluded this was too variable to track changes in individual players [40]. Our findings relating to HSR are somewhat unsurprising given the high variability demonstrated in competitive games across a season in professional soccer [41]. It has been proposed that the individualisation of HSR thresholds according to individual fitness characteristics may provide more stability in capturing running performance [41,42]. The current findings, however, demonstrate measures of HSRr provided no improved reliability during SSGs than HSRa distance. This finding is supported by Scott and Lovell [43] who found the individualisation of speed thresholds, using both maximal sprint speed and maximal aerobic speed, did not enhance the dose-response measurement of internal training load. Taken together, these findings question the efficacy of practitioners dedicating the additional time and resources (i.e., specialist laboratory and field-based testing) necessary to establish individualised speed thresholds. Speed and distance metrics, such as HSR, have been suggested to neglect the energetically taxing changes in velocity in different planes of motion associated within soccer activities [20]. Tri-axial accelerometer-derived variables measuring movement in three planes, such as PL, have been observed as an alternative to monitor intermittent multi-directional activities such as soccer [19,20]. Within the current study, PL had a SNR >1, suggesting that it can be used to detect meaningful differences within a specific group (i.e., team) of players. However, this was not the case when PL was made relative to time (PL/min) or distance covered (PL/metre). On an individual level, there was lower within-subject variation across PL and PL/min. Some of this variability may be extenuated to the placement of the device (between the scapulae), with suggestions that foot-mounted inertial sensors may be a more appropriate method to capture these specific movements [44]. Still, PL has been linked to alterations in acute fatigue within a fixed soccer simulation, postulating this may be able to detect alterations within an individual’s locomotor efficiency [29]. Links have been made between PL/min and the neuromuscular fatigue levels of Australian Football players [45]. Furthermore, Rowell and colleagues demonstrated reductions in PL/min during a standardised training game were associated with the same reductions in subsequent match external load metrics as those measured when countermovement jump performance (flight time: contact time) was reduced [11]. Given that such reductions had the same implications for match exercise intensity, the authors of that study postulated that PL/min could be used to assess fatigue in situ via a standardised training game, without the burden of additional jump testing [11]. Our findings, however, suggest that caution may be needed when monitoring certain PL variables during training games, due to the higher within- and between-subject variability witnessed. Nevertheless, it appears the accumulation of the PL may be highly individualised and further work is warranted to investigate if such variability is driven by fatigue and subsequent changes to movement patterns. In addition to prior findings linking PL and PL/min to markers of fatigue, changes to the within-match contribution of individual planes of PL have been shown in both soccer match play and a simulation as a possible means for assessing fatigue in-situ [11,20,29]. In the current study, low CV%, both within- and between-subjects, were observed for the percentage contribution of individual component planes to PL. Reductions in PLV (%) have been shown during elite Australian Football matches when the players started the game with existing neuromuscular fatigue [19]. However, individual variability (unclear results) was demonstrated in PLV (%) during soccer match play in those experiencing fatigue [45]. In addition, associations have been made between the increased contribution of PLML and the level of neuromuscular fatigue, assessed via both a jump test and a standardised training game [11]. Evidence of fatigue-induced changes in the relative contributions of the individual component planes to the vector magnitude is of interest given the low variability of these metrics demonstrated in this study. It would therefore seem pertinent to investigate the potential association between changes in fatigue-induced movement strategies and sensitivity of the individual component planes to detect these within training games in future work. Understanding the meaningful difference of external training load measures within an individual’s response to a given activity can help practitioners with planning and intervention strategies for the individual (e.g., training loads, recovery interventions) [11,20,38]. Taken together, it is apparent that global measures of ‘volume’ (e.g. TD, PL) present more stable measures of external training load to monitor during training games between individual players. These values however may be of less interest to practitioners than relative (per minute and metre) and higher-intensity metrics, due to the use of the latter in fatigue assessments [11], return to play [23] and physical conditioning [24] strategies. Based on the findings of the current study, we suggest caution is taken when using such volume and intensity metrics to plan and monitor external training loads; instead, practitioners may consider implementing other controlled training and assessment tools alongside standardised training drills, particularly when fatigue monitoring is of focus. Additionally, practitioners may look to use ‘live’ monitoring and feedback to communicate an individual’s actual external load compared to that which was planned, when using drills as part of conditioning and rehabilitation programming to appropriate track and, where suitable, adjust individual training loads. The current study demonstrates a process that practitioners can undertake to better understand the variation in external training load measures across trials of a standardised training game in their own setting. Findings of the current study highlights the individual variation within a team sport setting that should be considered and assessed, especially if games are prescribed with specific external training load objectives. Some limitations of this study should also be noted. Although incorporating two teams increased the data available to analyse, the potential effect of different playing level (first and second tier) on the results is unknown. The subjects were professional male footballers and so these findings may not be replicated in other populations, such as female footballers or youth players. The training drill formats used were based on those used in each applied setting, but many other designs are frequently used (i.e., number of players, pitch size, drill rules etc.). In addition, data pertaining to prior training load and fatigue status were not collected in the current study. Future work should explore what drives this variability, how these variables are influenced by prior training load and/or fatigue, and whether these metrics are sensitive to changes in movement patterns, to provide greater insight into their capacity to be used within testing and training strategies. Conclusions This study suggests the percentage contribution of the individual planes to PL demonstrated good reliability, both within- and between-subjects. Good reliability was observed for TD, m/min, PL, and PL/min on a group level but high within-subject variation was demonstrated for these variables. Therefore, it is recommended to monitor these external training load variables on an individual level. Both HSRa and HSRr showed poor reliability. These findings suggest that practitioners can use TD, m/min, PL, and PL/min to detect meaningful group differences in the external load of standardised training games using the current formats. However, HSR may not be an appropriate metric for detecting meaningful changes. The variability in the external training load measures observed in this study demonstrates the need for greater attention on physical outcomes when prescribing games in soccer training. These findings also question the transferability of some of the research into training games (including small, medium, and large sided games) variation into the applied environment if only between-subject observations have been made. Awareness of this variability, along with monitoring of such metrics on an individual level, may assist practitioners with a better understanding of the external loads of training games in the applied environment.

Alkhodari, Mohanad;Khandoker, Ahsan H.

doi: 10.1371/journal.pone.0262448pmid: 35025945

Introduction Corona virus 2019 (COVID-19), which is a novel pathogen of the severe acute respiratory syndrome coronavirus 2 (SARS-Cov-2), appeared first in late November 2019 and ever since, it has caused a global epidemic problem by spreading all over the world [1]. According to the world heath organization (WHO) April 2021 report [2], there have been nearly 150 million confirmed cases and over 3 million deaths since the pandemic broke out in 2019. Additionally, the United States (US) have reported the highest number of cumulative cases and deaths with over 32.5 million and 500,000, respectively. These huge numbers have caused many healthcare services to be severely burdened especially with the ability of the virus to develop more genomic variants and spread more readily among people. India, which is one of the world’s biggest suppliers of vaccines, is now severely suffering from the pandemic after the explosion of cases due to a new variant of COVID-19. It has reached more than 17.5 million confirmed cases, setting it behind the US as the second worst hit country [2, 3]. COVID-19 patients usually range from being asymptomatic to developing pneumonia and in severe cases, death. In most reported cases, the virus remains incubation for a period of 1 to 14 days before the symptoms of an infection start arising [4]. Patients carrying COVID-19 have exhibited common signs and symptoms including cough, shortness of breath, fever, fatigue, and other acute respiratory distress syndromes (ARDS) [5, 6]. Most infected people suffer from mild to moderate viral symptoms, however, they end up by being recovered. On the other hand, patients who develop severe symptoms such as severe pneumonia are mostly people over 60 years of age with conditions such as diabetes, cardiovascular diseases (CVD), hypertension, and cancer [4, 5]. On most cases, the early diagnosis of COVID-19 helps in preventing its spreading and development to severe infection stages. This is usually done by following steps of early patient isolation and contact tracing. Furthermore, timely medication and efficient treatment reduces symptoms and results in lowering the mortality rate of this pandemic [7]. The current gold standard in diagnosing COVID-19 is the reverse-transcription polymerase chain reaction (RT-PCR) assay [8, 9]. It is the most commonly used technique worldwide to successfully confirm the existence of this viral infection. Additionally, examinations of the ribonucleic acid (RNA) in patients carrying the virus provide further information about the infection, however, it requires longer time for diagnosis and is not considered as accurate as other diagnostic techniques [10]. The integration of computed tomography (CT) screening (X-ray radiations) is another effective diagnostic tool (sensitivity ≥90%) that often provides supplemental information about the severity and progression of COVID-19 in lungs [11, 12]. CT imaging is not recommended for patients at the early stages of the infection, i.e., showing asymptomatic to mild symptoms. It provides useful details about the lungs in patients with moderate to severe stages due to the disturbances in pulmonary tissues and its corresponding functions [13]. Most recently, several studies have utilized the new emerging algorithms in artificial intelligence (AI) to detect and classify COVID-19 in CT and X-ray images [14]. Machine and deep learning algorithms were implemented in several studies (taking CT images as inputs) with a discrimination accuracy reaching over 95% between healthy and infected subjects [15–20]. The major contribution of these studies is the ability of trained models including support vector machine (SVM) and convolutional neural networks (CNN) in detecting COVID-19 in CT images with minimal pre-processing steps. Moreover, several studies have utilized deep learning with extra feature fusion techniques and entropy-controlled optimization [21], rank-based average pooling [22], pseudo-Zernike moment (PZM) [23], and internet-of-things [24] to detect COVID-19 in CT images. In addition, there has been extensive research carried out for COVID-19 assessment using X-ray images and machine learning [25–27]. Majority of the current state-of-art approaches rely on two-dimensional (2D) X-ray images of the lungs to train neural networks on extracting features and thus identifying subjects carrying the viral infection. Despite of the high levels of accuracy achieved in most of the studies, CT and X-ray imaging use ionizing radiations that make them not feasible for frequent testing. In addition, these imaging modalities may not be available in all public healthcare services, especially for countries who are swamped with the pandemic, due to their costs and additional maintenance requirements. Most recently, researchers have utilized a safer and simpler imaging approach based on ultrasound to screen lungs for COVID-19 [28] and achieved high levels of performance (accuracy > 89%). Therefore, finding promising alternatives that are simple, fast, and cost-effective is an ultimate goal to researchers when it comes to integrating these techniques with machine learning. Biological respiratory signals, such as coughing and breathing sounds, could be another promising tool to indicate the existence of the viral infection [29], as these signals have a direct connection with lungs. Respiratory auscultation is considered as a safe and non-invasive technique to diagnose the respiratory system and its associated organs. This technique is usually done by clinicians using an electronic stethoscope to hear and record the air sound moving inside and outside lungs while breathing or coughing. Thus, an indication of any pulmonary anomalies could be detected and identified [30–32]. Due to the simplicity in recording respiratory signals, lung sounds could carry useful information about the viral infection, and thus, could set an early alert to the patient before moving on with further and extensive medication procedures. In addition, combining the simple respiratory signals with AI algorithms could be a key to enhance the sensitivity of detection for positive cases due to its ability to generalize over a wide set of data with less computational complexity [33]. Many studies have investigated the information carried by respiratory sounds in patients tested positive for COVID-19 [34–36]. Furthermore, it has been found that vocal patterns extracted from COVID-19 patients’ speech recordings carry indicative biomarkers for the existence of the viral infection [37]. In addition, a telemedicine approach was also explored to observe evidences on the sequential changes in respiratory sounds as a result of COVID-19 infection [38]. Most recently, AI was utilized in one study to recognize COVID-19 in cough signals [39] and in another to evaluate the severity of patients’ illness, sleep quality, fatigue, and anxiety through speech recordings [40]. Despite of the high levels of performance achieved in the aforementioned AI-based studies, further investigations on the capability of respiratory sounds in carrying useful information about COVID-19 are still required, especially when embedded within the framework of sophisticated AI-based algorithms. Furthermore, due to the explosion in the number of confirmed positive COVID-19 cases all over the world, it is essential to ensure providing a system capable of recognizing the disease in signals recording through portable devices, such as computers or smartphones, instead of regular clinic-based electronic stethoscopes. Motivated by the aforementioned, a complete deep learning approach is proposed in this paper for a successful detection of COVID-19 using only breathing sounds recorded through a microphone of a smartphone device (Fig 1). The proposed approach serves as a rapid, no-cost, and easily distributed pre-screening tool for COVID-19, especially for countries who are in a complete lockdown due to the wide spread of the pandemic. Although the current gold standard, RT-PCR, provides high success rates in detecting the viral infection, it has various limitations including the high expenses involved with equipment and chemical agents, requirement of expert nurses and doctors for diagnosis, violation of social distancing, and the long testing time required to obtain results (2-3 days). Thus, the development of a deep learning model overcomes most of these limitations and allows for a better revival in the healthcare and economic sectors in several countries. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. A graphical abstract of the complete procedure followed in this study. The input data includes breathing sounds collected from an open-access database for respiratory sounds (Coswara [41]) recorded via smartphone microphone. The data includes a total of 240 participants, out of which 120 subjects were suffering from COVID-19, while the remaining 120 were healthy (control group). A deep learning framework was then utilized based on hand-crafted features extracted by feature engineering techniques, as well as deep-activated features extracted by a combination of convolutional and recurrent neural network. The performance was then evaluated and further discussed on the use of artificial intelligence (AI) as a successful pre-screening tool for COVID-19. https://doi.org/10.1371/journal.pone.0262448.g001 Furthermore, the novelty of this work lies in utilizing smartphone-based breathing recordings within this deep learning model, which, when compared to conventional respiratory auscultation devices, i.e., electronic stethoscopes, are more preferable due to their higher accessibility by wider population. This plays an important factor in obtaining medical information about COVID-19 patients in a timely manner while at the same time maintaining an isolated behaviour between people. Additionally, this study covers patients who are mostly from India, which is severely suffering from a new genomic variant (first reported in December 2020) of COVID-19 capable of escaping the immune system and most of the available vaccines [2, 42]. Thus, it gives an insight on the ability of AI algorithms in detecting this viral infection in patients carrying this new variant, including asymptomatic. Lastly, the study presented herein investigates signal characteristics contaminated within shallow and deep breathing sounds of COVID-19 and healthy subjects through deep-activated attributes (neural network activations) of the original signals as well as wide attributes (hand-crafted features) of the signals and their corresponding mel-frequency cepstrum (MFC). The utilization of one-dimensional (1D) signals within a successful deep learning framework allows for a simple, yet effective, AI design that does not require heavy memory requirements. This serves as a suitable solution for further development of telemedicine and smartphone applications for COVID-19 (or other pandemics) that can provide real-time results and communications between patients and clinicians in an efficient and timely manner. Therefore, as a pre-screening tool for COVID-19, this allows for a better and faster isolation and contact tracing than currently available techniques. Materials and methods Dataset collection and subjects information The dataset used in this study was obtained from Coswara [41], which is a project aiming towards providing an open-access database for respiratory sounds of healthy and unhealthy individuals, including those suffering from COVID-19. The project is a worldwide respiratory data collection effort that was first initiated in August, 7th 2020. Ever since, it has collected data from more than 1,600 participants (Male: 1185, Female: 415) from allover the world (mostly Indian population). The database was approved by the Indian institute of science (IISc), human ethics committee, Bangalore, India, and conforms to the ethical principles outlined in the declaration of Helsinki. No personally identifiable information about participants was collected and the participants’ data was fully anonymized during storage in the database. The database includes breath, cough, and voice sounds acquired via crowdsourcing using an interactive website application that was built for smartphone devices [43]. The average interaction time with the application was 5-7 minutes. All sounds were recorded using the microphone of a smartphone and sampled with a sampling frequency of 48 kHz. The participants had the freedom to select any device for recording their respiratory sounds, which reduces device-specific bias in the data. The audio samples (stored in. WAV format) for all participants were manually curated through a web interface that allows multiple annotators to go through each audio file and verify the quality as well as the correctness of labeling. All participants were requested to keep a 10 cm distance between the face and the device before starting the recording. So far, the database had a COVID-19 participants’ count of 120, which is almost 1-10 ratio to healthy (control) participants. In this study, all COVID-19 participants’ data was used, and the same number of samples from the control participants’ data was randomly selected to ensure a balanced dataset. Therefore, the dataset used in this study had a total of 240 subjects (COVID-19: 120, Control: 120). Furthermore, only breathing sounds of two types, namely shallow and deep, were obtained from every subject and used for further analysis. Figs 2 and 3 show examples from the shallow and deep breathing datasets, respectively, with their corresponding spectrogram representation. To ensure the inclusion of maximum information from each breathing recording as well as to cover at least 2-4 breathing cycles (inhale and exhale), a total of 16 seconds were considered, as the normal breathing pattern in adults ranges between 12 to 18 breaths per minute [44]. All recordings less than 16 seconds were padded with zeros. Furthermore, the final signals were resampled with a sampling frequency of 4 kHz. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Examples from the shallow breathing sounds recorded via smartphone microphone along with their corresponding spectrograms. Showing: (a-c) COVID-19 subjects (asymptomatic, mild, moderate), (d-f) healthy subjects. https://doi.org/10.1371/journal.pone.0262448.g002 Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Examples from the deep breathing sounds recorded via smartphone microphone along with their corresponding spectrograms. Showing: (a-c) COVID-19 subjects (asymptomatic, mild, moderate), (d-f) healthy subjects. https://doi.org/10.1371/journal.pone.0262448.g003 The demographic and clinical information of the selected subjects is provided in Table 1. All values are provided as range and mean±std (age), numbers (sex), and yes/no (1/0). To check for most significant variables, a linear regression fitting algorithm [45] was applied. An indication of a significant difference between COVID-19 and healthy subjects was obtained whenever the p-value was less than 0.05 (bold). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. The demographic and clinical information of COVID-19 and healthy (control) subjects included in the study. https://doi.org/10.1371/journal.pone.0262448.t001 Deep learning framework The deep learning framework proposed in this study (Fig 4) includes a combination of hand-crafted features as well as deep-activated features learned through model’s training and reflected as time-activations of the input. To extract hand-crafted features, various algorithm and functions were used to obtain signal attributes from the original breathing recording and from its corresponding mel-frequency cepstral coefficients (MFCC). In addition, deep-activated learned features were obtained from the original breathing recording through a combined neural network that consists of convolutional and recurrent neural networks. Each part of this framework is briefly described in the following subsections and illustrated as a pseudocode in 1. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. The framework of deep learning followed in this study. The framework includes a combination of hand-crafted features and deep-activated features. Deep features were obtained through a combined convolutional and recurrent neural network (CNN-BiLSTM), and the final classification layer uses both features sets to discriminate between COVID-19 and healthy subjects. https://doi.org/10.1371/journal.pone.0262448.g004 Algorithm 1 Training Deep Learning Model for COVID-19 Prediction Input: 120 COVID-19 / 120 healthy breathing recordings (shallow or deep) Output: Trained model to predict COVID-19 or healthy 1: for Every breathing recording do 2:  Calculate kurtosis k (Eq 1) and skewness s (Eq 2) 3:  Calculate eample entropy SampEn (Eq 3) and spectral entropy SE (Eq 4) 4:  Calculate fractal dimensions—Higuchi HFD (Eq 5) and Katz KFD (Eq 6) 5:  Calculate zero-crossing rate ZCR (Eq 7) 6:  Extract 13 Mel-frequency cepstral coefficients (MFCC) 7:  for Every MFCC signal do 8: Extract all aforementioned hand-crafted features 9:  end for 10: end for 11: χ2-test: All features plus age and sex, select best 20 features 12: for Every breathing recording do 13:  Apply volume control and time shift augmentation (24 augmented signals) 14: end for 15: Train CNN-BiLSTM network to extract deep-activated features from breathing recordings 16: Update the trained model using the 20 best hand-crafted features Hand-crafted features. These features refer to signal attributes that are extracted manually through various algorithms and functions in a process called feature engineering [46]. The advantage of following such process is that it can extract internal and hidden information within input data, i.e., sounds, and represent it as single or multiple values [47]. Thus, additional knowledge about the input data can be obtained and used for further analysis and evaluation. Hand-crafted features were initially extracted from the original breathing recordings, then, they were also extracted from the MFCC transformation of the signals. The features included in this study are, Kurtosis and skewness: In statistics, kurtosis is a quantification measure for the degree of extremity included within the tails of a distribution relative to the tails of a normal distribution. The more the distribution is outlier-prone, the higher the kurtosis values, and vice-versa. A kurtosis of 3 indicates that the values follow a normal distribution. On the other hand, skewness is a measure for the asymmetry of the data that deviates it from the mean of the normal distribution. If the skewness is negative, then the data are more spread towards the left side of the mean, while a positive skewness indicates data spreading towards the right side of the mean [48]. A skewness of zero indicates that the values follow a normal distribution. Kurtosis (k) and skewness (s) can be calculated as, (1) (2) where X included input values, μ and σ are the mean and standard deviation values of the input, respectively, and E is an expectation operator. Sample entropy: In physiological signals, the sample entropy (SampEn) provides a measure for complexity contaminated within time sequences. This feature represent the randomness contaminated within a signal by embedding it into a phase space to estimate the increment rate in the number of phase space patterns. It can be calculated though the negative natural logarithm of a probability that segments of length m match their consecutive segments under a value of tolerance (r) [49] as follows, (3) where segmentA is the first segment in the time sequence and segmentA+ 1 is the consecutive segment. Spectral entropy: To measure time series irregularity, spectral entropy (SE) provides a frequency domain entropy measure as a sum of the normalize signal spectral power [50]. It differs from the aforementioned SampEN in analyzing the frequency spectrum of signals rather than time sequences and phase. Based on Shannon’s entropy, the SE can be calculated as, (4) where N is the total number of frequency points and P(n) is the probability distribution of the power spectrum. Fractal dimension. Higuchi and Katz [51, 52] provided two methods to measure statistically the complexity in a time series. More specifically, fractal dimension measures provide an index for characterizing how much a time series is self-similar over some region of space. Higuchi (HFD) and Katz (KFD) fractal dimensions can be calculated as, (5) (6) where L(k) is the length of the fractal curve, r is the selected time interval, N is the length of the signal, and d is the maximum distance between an initial point to other points. Zero-crossing rate. To measure the number of times a signal has passed through the zero point, a zero-crossing rate (ZCR) measure is provided. In other words, ZCR refers to the rate of sign-changes in the signals’ data points. It can be calculated as follows, (7) where xt = 1 if the signal has a positive value at time step t and a value of 0 otherwise. Mel-frequency cepstral coefficients (MFCC). To better represent speech and voice signals, MFCC provides a set of coefficients of the discrete cosine transformed (DCT) logarithm of a signal’s spectrum (mel-frequency cepstrum (MFC)). It is considered as an overall representation of the information contaminated within signals regarding the changes in its different spectrum bands [53, 54]. Briefly, to obtain the coefficients, the signals goes through several steps, namely windowing the signal, applying discrete Fourier transform (DFT), calculating the log energy of the magnitude, transforming the frequencies to the Mel-scale, and applying inverse DCT. In this work, 13 coefficients (MFCC-1 to MFCC-13) were obtained from each breathing sound signal. For every coefficient, the aforementioned features were extracted and stored as an additional MFCC hand-crafted features alongside the original breathing signals features. Deep-activated features. These features refer to attributes extracted from signals through a deep learning process and not by manual feature engineering techniques. The utilization of deep learning allows for the acquisition of optimized features extracted through deep convolutional layers about the structural information contaminated within signals. Furthermore, it has the ability to acquire the temporal (time changes) information carried through time sequences [55–57]. Such optimized features can be considered as a complete representation of the input data generated iteratively through an automated learning process. To achieve this, we used an advanced neural network based on a combination of convolutional neural network and bi-directional long short-term memory (CNN-BiLSTM). Neural network architecture. The structure of the network starts by 1D convolutional layers. In deep learning, convolutions refer to a multiple number of dot products applied to 1D signals on pre-defined segments. By applying consecutive convolutions, the network extracts deep attributes (activations) to form an overall feature map for the input data [56]. A single convolution on an input , where n is the total number of points, is usually calculated as, (8) where l is the layer index, h is the activation function, b is the bias of the jth feature map, M is the kernel size, is the weight of the jth feature map and mth filter index. In this work, three convolutional layers were used to form the first stage of the deep neural network. The kernel sizes of each layer are [9, 1], [5, 1], and [3, 1], respectively. Furthermore, the number of filters increases as the network becomes deeper, that is 16, 32, and 64, respectively. Each convolutional layer was followed by a max-pooling layer to reduce the dimensionality as well as the complexity in the model. The max-pooling kernel size decreases as the network gets deeper with a [8, 1], [4, 1], and [2, 1] kernels for the three max-pooling layers, respectively. It is worth noting that each max-pooling layer was followed by a batch normalization (BN) layer to normalize all filters as well as by a rectified linear unit (ReLU) layer to set all values less than zero in the feature map to zero. The complete structure is illustrated in Fig 4. The network continues with additional extraction of temporal features through bi-directional LSTM units. In recurrent neural networks, LSTM units allows for the detection of long short-term dependencies between time sequence data points. Thus, it overcomes the issues of exploding and vanishing gradients in chain-like structures during training [55, 58]. An LSTM block includes a collection of gates, namely input (i), output (o), and forget (f) gates. These gates handle the flow of data as well as the processing of the input and output activations within the network’s memory. The information of the main cell (Ct) at any instance (t) within the block can be calculated as, (9) where ct is the input to the main cell and Ct−1 includes the information at the previous time instance. In addition, the network performs hidden units (ht) activations on the output and main cell input using a sigmoid function as follows, (10) Furthermore, a bi-drectional functionality (BiLSTM) allows the network to process data in both the forward and backward direction as follows, (11) where and are the outputs of the hidden layers in the forward and backward directions, respectively, for all N levels of stack and by is a bias vector. In this work, a BiLSTM hidden units functionality was selected with a total number of hidden units of 256. Thus, the resulting output is a 512 vector (both directions) of the extracted hidden units of every input. BiLSTM activations. To be able to utilize the parameters that the BiLSTM units have learned, the activations that correspond to each hidden unit were extracted from the network for each input signal. Recurrent neural network activations of a pre-trained network are vectors that carry the final learned attributes about different time steps within the input [59, 60]. In this work, these activations were the final signal attributes extracted from each input signal. Such attributes are referred to as deep-activated features in this work (Fig 4). To compute these activations, we use function activations() in MATLAB inputting the trained network, the selected data (breathing recording), and the chosen features layer (BiLSTM). Furthermore, they were concatenated with the hand-crafted features alongside age and sex information and used for the final predictions by the network. Network configuration and training scheme. Prior to deep learning model training, several data preparation and network fine-tuning steps were followed including data augmentation, best features selection, deciding the training and testing scheme, and network parameters configuration. Data augmentation. Due to the small sample size available, it is critical for deep learning applications to include augmented data. Instead of training the model on the existing dataset only, data augmentation allows for the generation of new modified copies of the original samples. These new copies have similar characteristics of the original data, however, they are slightly adjusted as if they are coming from a new source (subject). Such procedure is essential to expose the deep learning model to more variations in the training data. Thus, making it robust and less biased when attempting to generalize the parameters on new data [61]. Furthermore, it was essential to prevent the model from over-fitting, where the model learns exactly the input data only with a very minimal generalization capabilities for unseen data [62]. In this study, 3,000 samples per class were generated using two 1D data augmentation techniques as follows, Volume control: Adjusts the strength of signals in decibels (dB) for the generated data [63] with a probability of 0.8 and gain ranging between -5 and 5 dB. Time shift: Modifies time steps of the signals to illustrate shifting in time for the generated data [64] with a shifting range of [-0.005 to 0.005] seconds. Best features selection. To ensure the inclusion of the most important hand-crafted features within the trained model, a statistical univariate chi-square test (χ2-test) was applied. In this test, a feature is decided to be important if the observed statistical analysis using this feature matches with the expected one, i.e., label [65]. Furthermore, an important feature indicates that it is considered significant in discriminating between two categories with a p-value < 0.05. The lower the p-value, the more the feature is dependent on the category label. The importance score can then be calculated as, (12) In this work, hand-crafted features extracted from the original breathing signals and from the MFCC alongside the age and sex information were selected for this test. The best 20 features were included in the final best features vector within the final fully-connected layer (along with the deep-activated features) for predictions. Training configuration. To ensure the inclusion of the whole available data, a leave-one-out training and testing scheme was followed. In this scheme, a total of 240 iterations (number of input samples) were applied, where in each iteration, an ith subject was used as the testing subject, and the remaining subjects were used for model’s training. This scheme was essential to be followed to provide a prediction for each subject in the dataset. Furthermore, the network was optimized using adaptive moment estimation (ADAM) solver [66] and with a learning rate of 0.001. The L2-regularization was set to 106 and the mini-batch size to 32. Performance evaluation The performance of the proposed deep learning model in discriminating COVID-19 from healthy subjects was evaluated using traditional evaluation metrics including accuracy, sensitivity, specificity, precision, and F1-score. Additionally, the area under the receiver operating characteristic (AUROC) curves was analysed for each category to show the true positive rate (TPR) versus the false positive rate (FPR) [67]. Dataset collection and subjects information The dataset used in this study was obtained from Coswara [41], which is a project aiming towards providing an open-access database for respiratory sounds of healthy and unhealthy individuals, including those suffering from COVID-19. The project is a worldwide respiratory data collection effort that was first initiated in August, 7th 2020. Ever since, it has collected data from more than 1,600 participants (Male: 1185, Female: 415) from allover the world (mostly Indian population). The database was approved by the Indian institute of science (IISc), human ethics committee, Bangalore, India, and conforms to the ethical principles outlined in the declaration of Helsinki. No personally identifiable information about participants was collected and the participants’ data was fully anonymized during storage in the database. The database includes breath, cough, and voice sounds acquired via crowdsourcing using an interactive website application that was built for smartphone devices [43]. The average interaction time with the application was 5-7 minutes. All sounds were recorded using the microphone of a smartphone and sampled with a sampling frequency of 48 kHz. The participants had the freedom to select any device for recording their respiratory sounds, which reduces device-specific bias in the data. The audio samples (stored in. WAV format) for all participants were manually curated through a web interface that allows multiple annotators to go through each audio file and verify the quality as well as the correctness of labeling. All participants were requested to keep a 10 cm distance between the face and the device before starting the recording. So far, the database had a COVID-19 participants’ count of 120, which is almost 1-10 ratio to healthy (control) participants. In this study, all COVID-19 participants’ data was used, and the same number of samples from the control participants’ data was randomly selected to ensure a balanced dataset. Therefore, the dataset used in this study had a total of 240 subjects (COVID-19: 120, Control: 120). Furthermore, only breathing sounds of two types, namely shallow and deep, were obtained from every subject and used for further analysis. Figs 2 and 3 show examples from the shallow and deep breathing datasets, respectively, with their corresponding spectrogram representation. To ensure the inclusion of maximum information from each breathing recording as well as to cover at least 2-4 breathing cycles (inhale and exhale), a total of 16 seconds were considered, as the normal breathing pattern in adults ranges between 12 to 18 breaths per minute [44]. All recordings less than 16 seconds were padded with zeros. Furthermore, the final signals were resampled with a sampling frequency of 4 kHz. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Examples from the shallow breathing sounds recorded via smartphone microphone along with their corresponding spectrograms. Showing: (a-c) COVID-19 subjects (asymptomatic, mild, moderate), (d-f) healthy subjects. https://doi.org/10.1371/journal.pone.0262448.g002 Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Examples from the deep breathing sounds recorded via smartphone microphone along with their corresponding spectrograms. Showing: (a-c) COVID-19 subjects (asymptomatic, mild, moderate), (d-f) healthy subjects. https://doi.org/10.1371/journal.pone.0262448.g003 The demographic and clinical information of the selected subjects is provided in Table 1. All values are provided as range and mean±std (age), numbers (sex), and yes/no (1/0). To check for most significant variables, a linear regression fitting algorithm [45] was applied. An indication of a significant difference between COVID-19 and healthy subjects was obtained whenever the p-value was less than 0.05 (bold). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. The demographic and clinical information of COVID-19 and healthy (control) subjects included in the study. https://doi.org/10.1371/journal.pone.0262448.t001 Deep learning framework The deep learning framework proposed in this study (Fig 4) includes a combination of hand-crafted features as well as deep-activated features learned through model’s training and reflected as time-activations of the input. To extract hand-crafted features, various algorithm and functions were used to obtain signal attributes from the original breathing recording and from its corresponding mel-frequency cepstral coefficients (MFCC). In addition, deep-activated learned features were obtained from the original breathing recording through a combined neural network that consists of convolutional and recurrent neural networks. Each part of this framework is briefly described in the following subsections and illustrated as a pseudocode in 1. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. The framework of deep learning followed in this study. The framework includes a combination of hand-crafted features and deep-activated features. Deep features were obtained through a combined convolutional and recurrent neural network (CNN-BiLSTM), and the final classification layer uses both features sets to discriminate between COVID-19 and healthy subjects. https://doi.org/10.1371/journal.pone.0262448.g004 Algorithm 1 Training Deep Learning Model for COVID-19 Prediction Input: 120 COVID-19 / 120 healthy breathing recordings (shallow or deep) Output: Trained model to predict COVID-19 or healthy 1: for Every breathing recording do 2:  Calculate kurtosis k (Eq 1) and skewness s (Eq 2) 3:  Calculate eample entropy SampEn (Eq 3) and spectral entropy SE (Eq 4) 4:  Calculate fractal dimensions—Higuchi HFD (Eq 5) and Katz KFD (Eq 6) 5:  Calculate zero-crossing rate ZCR (Eq 7) 6:  Extract 13 Mel-frequency cepstral coefficients (MFCC) 7:  for Every MFCC signal do 8: Extract all aforementioned hand-crafted features 9:  end for 10: end for 11: χ2-test: All features plus age and sex, select best 20 features 12: for Every breathing recording do 13:  Apply volume control and time shift augmentation (24 augmented signals) 14: end for 15: Train CNN-BiLSTM network to extract deep-activated features from breathing recordings 16: Update the trained model using the 20 best hand-crafted features Hand-crafted features. These features refer to signal attributes that are extracted manually through various algorithms and functions in a process called feature engineering [46]. The advantage of following such process is that it can extract internal and hidden information within input data, i.e., sounds, and represent it as single or multiple values [47]. Thus, additional knowledge about the input data can be obtained and used for further analysis and evaluation. Hand-crafted features were initially extracted from the original breathing recordings, then, they were also extracted from the MFCC transformation of the signals. The features included in this study are, Kurtosis and skewness: In statistics, kurtosis is a quantification measure for the degree of extremity included within the tails of a distribution relative to the tails of a normal distribution. The more the distribution is outlier-prone, the higher the kurtosis values, and vice-versa. A kurtosis of 3 indicates that the values follow a normal distribution. On the other hand, skewness is a measure for the asymmetry of the data that deviates it from the mean of the normal distribution. If the skewness is negative, then the data are more spread towards the left side of the mean, while a positive skewness indicates data spreading towards the right side of the mean [48]. A skewness of zero indicates that the values follow a normal distribution. Kurtosis (k) and skewness (s) can be calculated as, (1) (2) where X included input values, μ and σ are the mean and standard deviation values of the input, respectively, and E is an expectation operator. Sample entropy: In physiological signals, the sample entropy (SampEn) provides a measure for complexity contaminated within time sequences. This feature represent the randomness contaminated within a signal by embedding it into a phase space to estimate the increment rate in the number of phase space patterns. It can be calculated though the negative natural logarithm of a probability that segments of length m match their consecutive segments under a value of tolerance (r) [49] as follows, (3) where segmentA is the first segment in the time sequence and segmentA+ 1 is the consecutive segment. Spectral entropy: To measure time series irregularity, spectral entropy (SE) provides a frequency domain entropy measure as a sum of the normalize signal spectral power [50]. It differs from the aforementioned SampEN in analyzing the frequency spectrum of signals rather than time sequences and phase. Based on Shannon’s entropy, the SE can be calculated as, (4) where N is the total number of frequency points and P(n) is the probability distribution of the power spectrum. Fractal dimension. Higuchi and Katz [51, 52] provided two methods to measure statistically the complexity in a time series. More specifically, fractal dimension measures provide an index for characterizing how much a time series is self-similar over some region of space. Higuchi (HFD) and Katz (KFD) fractal dimensions can be calculated as, (5) (6) where L(k) is the length of the fractal curve, r is the selected time interval, N is the length of the signal, and d is the maximum distance between an initial point to other points. Zero-crossing rate. To measure the number of times a signal has passed through the zero point, a zero-crossing rate (ZCR) measure is provided. In other words, ZCR refers to the rate of sign-changes in the signals’ data points. It can be calculated as follows, (7) where xt = 1 if the signal has a positive value at time step t and a value of 0 otherwise. Mel-frequency cepstral coefficients (MFCC). To better represent speech and voice signals, MFCC provides a set of coefficients of the discrete cosine transformed (DCT) logarithm of a signal’s spectrum (mel-frequency cepstrum (MFC)). It is considered as an overall representation of the information contaminated within signals regarding the changes in its different spectrum bands [53, 54]. Briefly, to obtain the coefficients, the signals goes through several steps, namely windowing the signal, applying discrete Fourier transform (DFT), calculating the log energy of the magnitude, transforming the frequencies to the Mel-scale, and applying inverse DCT. In this work, 13 coefficients (MFCC-1 to MFCC-13) were obtained from each breathing sound signal. For every coefficient, the aforementioned features were extracted and stored as an additional MFCC hand-crafted features alongside the original breathing signals features. Deep-activated features. These features refer to attributes extracted from signals through a deep learning process and not by manual feature engineering techniques. The utilization of deep learning allows for the acquisition of optimized features extracted through deep convolutional layers about the structural information contaminated within signals. Furthermore, it has the ability to acquire the temporal (time changes) information carried through time sequences [55–57]. Such optimized features can be considered as a complete representation of the input data generated iteratively through an automated learning process. To achieve this, we used an advanced neural network based on a combination of convolutional neural network and bi-directional long short-term memory (CNN-BiLSTM). Neural network architecture. The structure of the network starts by 1D convolutional layers. In deep learning, convolutions refer to a multiple number of dot products applied to 1D signals on pre-defined segments. By applying consecutive convolutions, the network extracts deep attributes (activations) to form an overall feature map for the input data [56]. A single convolution on an input , where n is the total number of points, is usually calculated as, (8) where l is the layer index, h is the activation function, b is the bias of the jth feature map, M is the kernel size, is the weight of the jth feature map and mth filter index. In this work, three convolutional layers were used to form the first stage of the deep neural network. The kernel sizes of each layer are [9, 1], [5, 1], and [3, 1], respectively. Furthermore, the number of filters increases as the network becomes deeper, that is 16, 32, and 64, respectively. Each convolutional layer was followed by a max-pooling layer to reduce the dimensionality as well as the complexity in the model. The max-pooling kernel size decreases as the network gets deeper with a [8, 1], [4, 1], and [2, 1] kernels for the three max-pooling layers, respectively. It is worth noting that each max-pooling layer was followed by a batch normalization (BN) layer to normalize all filters as well as by a rectified linear unit (ReLU) layer to set all values less than zero in the feature map to zero. The complete structure is illustrated in Fig 4. The network continues with additional extraction of temporal features through bi-directional LSTM units. In recurrent neural networks, LSTM units allows for the detection of long short-term dependencies between time sequence data points. Thus, it overcomes the issues of exploding and vanishing gradients in chain-like structures during training [55, 58]. An LSTM block includes a collection of gates, namely input (i), output (o), and forget (f) gates. These gates handle the flow of data as well as the processing of the input and output activations within the network’s memory. The information of the main cell (Ct) at any instance (t) within the block can be calculated as, (9) where ct is the input to the main cell and Ct−1 includes the information at the previous time instance. In addition, the network performs hidden units (ht) activations on the output and main cell input using a sigmoid function as follows, (10) Furthermore, a bi-drectional functionality (BiLSTM) allows the network to process data in both the forward and backward direction as follows, (11) where and are the outputs of the hidden layers in the forward and backward directions, respectively, for all N levels of stack and by is a bias vector. In this work, a BiLSTM hidden units functionality was selected with a total number of hidden units of 256. Thus, the resulting output is a 512 vector (both directions) of the extracted hidden units of every input. BiLSTM activations. To be able to utilize the parameters that the BiLSTM units have learned, the activations that correspond to each hidden unit were extracted from the network for each input signal. Recurrent neural network activations of a pre-trained network are vectors that carry the final learned attributes about different time steps within the input [59, 60]. In this work, these activations were the final signal attributes extracted from each input signal. Such attributes are referred to as deep-activated features in this work (Fig 4). To compute these activations, we use function activations() in MATLAB inputting the trained network, the selected data (breathing recording), and the chosen features layer (BiLSTM). Furthermore, they were concatenated with the hand-crafted features alongside age and sex information and used for the final predictions by the network. Network configuration and training scheme. Prior to deep learning model training, several data preparation and network fine-tuning steps were followed including data augmentation, best features selection, deciding the training and testing scheme, and network parameters configuration. Data augmentation. Due to the small sample size available, it is critical for deep learning applications to include augmented data. Instead of training the model on the existing dataset only, data augmentation allows for the generation of new modified copies of the original samples. These new copies have similar characteristics of the original data, however, they are slightly adjusted as if they are coming from a new source (subject). Such procedure is essential to expose the deep learning model to more variations in the training data. Thus, making it robust and less biased when attempting to generalize the parameters on new data [61]. Furthermore, it was essential to prevent the model from over-fitting, where the model learns exactly the input data only with a very minimal generalization capabilities for unseen data [62]. In this study, 3,000 samples per class were generated using two 1D data augmentation techniques as follows, Volume control: Adjusts the strength of signals in decibels (dB) for the generated data [63] with a probability of 0.8 and gain ranging between -5 and 5 dB. Time shift: Modifies time steps of the signals to illustrate shifting in time for the generated data [64] with a shifting range of [-0.005 to 0.005] seconds. Best features selection. To ensure the inclusion of the most important hand-crafted features within the trained model, a statistical univariate chi-square test (χ2-test) was applied. In this test, a feature is decided to be important if the observed statistical analysis using this feature matches with the expected one, i.e., label [65]. Furthermore, an important feature indicates that it is considered significant in discriminating between two categories with a p-value < 0.05. The lower the p-value, the more the feature is dependent on the category label. The importance score can then be calculated as, (12) In this work, hand-crafted features extracted from the original breathing signals and from the MFCC alongside the age and sex information were selected for this test. The best 20 features were included in the final best features vector within the final fully-connected layer (along with the deep-activated features) for predictions. Training configuration. To ensure the inclusion of the whole available data, a leave-one-out training and testing scheme was followed. In this scheme, a total of 240 iterations (number of input samples) were applied, where in each iteration, an ith subject was used as the testing subject, and the remaining subjects were used for model’s training. This scheme was essential to be followed to provide a prediction for each subject in the dataset. Furthermore, the network was optimized using adaptive moment estimation (ADAM) solver [66] and with a learning rate of 0.001. The L2-regularization was set to 106 and the mini-batch size to 32. Hand-crafted features. These features refer to signal attributes that are extracted manually through various algorithms and functions in a process called feature engineering [46]. The advantage of following such process is that it can extract internal and hidden information within input data, i.e., sounds, and represent it as single or multiple values [47]. Thus, additional knowledge about the input data can be obtained and used for further analysis and evaluation. Hand-crafted features were initially extracted from the original breathing recordings, then, they were also extracted from the MFCC transformation of the signals. The features included in this study are, Kurtosis and skewness: In statistics, kurtosis is a quantification measure for the degree of extremity included within the tails of a distribution relative to the tails of a normal distribution. The more the distribution is outlier-prone, the higher the kurtosis values, and vice-versa. A kurtosis of 3 indicates that the values follow a normal distribution. On the other hand, skewness is a measure for the asymmetry of the data that deviates it from the mean of the normal distribution. If the skewness is negative, then the data are more spread towards the left side of the mean, while a positive skewness indicates data spreading towards the right side of the mean [48]. A skewness of zero indicates that the values follow a normal distribution. Kurtosis (k) and skewness (s) can be calculated as, (1) (2) where X included input values, μ and σ are the mean and standard deviation values of the input, respectively, and E is an expectation operator. Sample entropy: In physiological signals, the sample entropy (SampEn) provides a measure for complexity contaminated within time sequences. This feature represent the randomness contaminated within a signal by embedding it into a phase space to estimate the increment rate in the number of phase space patterns. It can be calculated though the negative natural logarithm of a probability that segments of length m match their consecutive segments under a value of tolerance (r) [49] as follows, (3) where segmentA is the first segment in the time sequence and segmentA+ 1 is the consecutive segment. Spectral entropy: To measure time series irregularity, spectral entropy (SE) provides a frequency domain entropy measure as a sum of the normalize signal spectral power [50]. It differs from the aforementioned SampEN in analyzing the frequency spectrum of signals rather than time sequences and phase. Based on Shannon’s entropy, the SE can be calculated as, (4) where N is the total number of frequency points and P(n) is the probability distribution of the power spectrum. Fractal dimension. Higuchi and Katz [51, 52] provided two methods to measure statistically the complexity in a time series. More specifically, fractal dimension measures provide an index for characterizing how much a time series is self-similar over some region of space. Higuchi (HFD) and Katz (KFD) fractal dimensions can be calculated as, (5) (6) where L(k) is the length of the fractal curve, r is the selected time interval, N is the length of the signal, and d is the maximum distance between an initial point to other points. Zero-crossing rate. To measure the number of times a signal has passed through the zero point, a zero-crossing rate (ZCR) measure is provided. In other words, ZCR refers to the rate of sign-changes in the signals’ data points. It can be calculated as follows, (7) where xt = 1 if the signal has a positive value at time step t and a value of 0 otherwise. Mel-frequency cepstral coefficients (MFCC). To better represent speech and voice signals, MFCC provides a set of coefficients of the discrete cosine transformed (DCT) logarithm of a signal’s spectrum (mel-frequency cepstrum (MFC)). It is considered as an overall representation of the information contaminated within signals regarding the changes in its different spectrum bands [53, 54]. Briefly, to obtain the coefficients, the signals goes through several steps, namely windowing the signal, applying discrete Fourier transform (DFT), calculating the log energy of the magnitude, transforming the frequencies to the Mel-scale, and applying inverse DCT. In this work, 13 coefficients (MFCC-1 to MFCC-13) were obtained from each breathing sound signal. For every coefficient, the aforementioned features were extracted and stored as an additional MFCC hand-crafted features alongside the original breathing signals features. Deep-activated features. These features refer to attributes extracted from signals through a deep learning process and not by manual feature engineering techniques. The utilization of deep learning allows for the acquisition of optimized features extracted through deep convolutional layers about the structural information contaminated within signals. Furthermore, it has the ability to acquire the temporal (time changes) information carried through time sequences [55–57]. Such optimized features can be considered as a complete representation of the input data generated iteratively through an automated learning process. To achieve this, we used an advanced neural network based on a combination of convolutional neural network and bi-directional long short-term memory (CNN-BiLSTM). Neural network architecture. The structure of the network starts by 1D convolutional layers. In deep learning, convolutions refer to a multiple number of dot products applied to 1D signals on pre-defined segments. By applying consecutive convolutions, the network extracts deep attributes (activations) to form an overall feature map for the input data [56]. A single convolution on an input , where n is the total number of points, is usually calculated as, (8) where l is the layer index, h is the activation function, b is the bias of the jth feature map, M is the kernel size, is the weight of the jth feature map and mth filter index. In this work, three convolutional layers were used to form the first stage of the deep neural network. The kernel sizes of each layer are [9, 1], [5, 1], and [3, 1], respectively. Furthermore, the number of filters increases as the network becomes deeper, that is 16, 32, and 64, respectively. Each convolutional layer was followed by a max-pooling layer to reduce the dimensionality as well as the complexity in the model. The max-pooling kernel size decreases as the network gets deeper with a [8, 1], [4, 1], and [2, 1] kernels for the three max-pooling layers, respectively. It is worth noting that each max-pooling layer was followed by a batch normalization (BN) layer to normalize all filters as well as by a rectified linear unit (ReLU) layer to set all values less than zero in the feature map to zero. The complete structure is illustrated in Fig 4. The network continues with additional extraction of temporal features through bi-directional LSTM units. In recurrent neural networks, LSTM units allows for the detection of long short-term dependencies between time sequence data points. Thus, it overcomes the issues of exploding and vanishing gradients in chain-like structures during training [55, 58]. An LSTM block includes a collection of gates, namely input (i), output (o), and forget (f) gates. These gates handle the flow of data as well as the processing of the input and output activations within the network’s memory. The information of the main cell (Ct) at any instance (t) within the block can be calculated as, (9) where ct is the input to the main cell and Ct−1 includes the information at the previous time instance. In addition, the network performs hidden units (ht) activations on the output and main cell input using a sigmoid function as follows, (10) Furthermore, a bi-drectional functionality (BiLSTM) allows the network to process data in both the forward and backward direction as follows, (11) where and are the outputs of the hidden layers in the forward and backward directions, respectively, for all N levels of stack and by is a bias vector. In this work, a BiLSTM hidden units functionality was selected with a total number of hidden units of 256. Thus, the resulting output is a 512 vector (both directions) of the extracted hidden units of every input. BiLSTM activations. To be able to utilize the parameters that the BiLSTM units have learned, the activations that correspond to each hidden unit were extracted from the network for each input signal. Recurrent neural network activations of a pre-trained network are vectors that carry the final learned attributes about different time steps within the input [59, 60]. In this work, these activations were the final signal attributes extracted from each input signal. Such attributes are referred to as deep-activated features in this work (Fig 4). To compute these activations, we use function activations() in MATLAB inputting the trained network, the selected data (breathing recording), and the chosen features layer (BiLSTM). Furthermore, they were concatenated with the hand-crafted features alongside age and sex information and used for the final predictions by the network. Network configuration and training scheme. Prior to deep learning model training, several data preparation and network fine-tuning steps were followed including data augmentation, best features selection, deciding the training and testing scheme, and network parameters configuration. Data augmentation. Due to the small sample size available, it is critical for deep learning applications to include augmented data. Instead of training the model on the existing dataset only, data augmentation allows for the generation of new modified copies of the original samples. These new copies have similar characteristics of the original data, however, they are slightly adjusted as if they are coming from a new source (subject). Such procedure is essential to expose the deep learning model to more variations in the training data. Thus, making it robust and less biased when attempting to generalize the parameters on new data [61]. Furthermore, it was essential to prevent the model from over-fitting, where the model learns exactly the input data only with a very minimal generalization capabilities for unseen data [62]. In this study, 3,000 samples per class were generated using two 1D data augmentation techniques as follows, Volume control: Adjusts the strength of signals in decibels (dB) for the generated data [63] with a probability of 0.8 and gain ranging between -5 and 5 dB. Time shift: Modifies time steps of the signals to illustrate shifting in time for the generated data [64] with a shifting range of [-0.005 to 0.005] seconds. Best features selection. To ensure the inclusion of the most important hand-crafted features within the trained model, a statistical univariate chi-square test (χ2-test) was applied. In this test, a feature is decided to be important if the observed statistical analysis using this feature matches with the expected one, i.e., label [65]. Furthermore, an important feature indicates that it is considered significant in discriminating between two categories with a p-value < 0.05. The lower the p-value, the more the feature is dependent on the category label. The importance score can then be calculated as, (12) In this work, hand-crafted features extracted from the original breathing signals and from the MFCC alongside the age and sex information were selected for this test. The best 20 features were included in the final best features vector within the final fully-connected layer (along with the deep-activated features) for predictions. Training configuration. To ensure the inclusion of the whole available data, a leave-one-out training and testing scheme was followed. In this scheme, a total of 240 iterations (number of input samples) were applied, where in each iteration, an ith subject was used as the testing subject, and the remaining subjects were used for model’s training. This scheme was essential to be followed to provide a prediction for each subject in the dataset. Furthermore, the network was optimized using adaptive moment estimation (ADAM) solver [66] and with a learning rate of 0.001. The L2-regularization was set to 106 and the mini-batch size to 32. Performance evaluation The performance of the proposed deep learning model in discriminating COVID-19 from healthy subjects was evaluated using traditional evaluation metrics including accuracy, sensitivity, specificity, precision, and F1-score. Additionally, the area under the receiver operating characteristic (AUROC) curves was analysed for each category to show the true positive rate (TPR) versus the false positive rate (FPR) [67]. Results Patient clinical information COVID-19 subjects included in this study had an average age of 34.04 years (± 13.45), while healthy subjects were slightly higher with an average of 36.02 years(±13.06). However, no significant difference was obtained for this variable across subjects (p = 0.149). It is worth noting that COVID-19 subjects with moderate conditions had a higher average age of 43.33 years. The distribution of male/female subjects across the two categories was close to 2:1 ratio with a majority of male subjects. Sex was not significantly different between COVID-19 and healthy subjects (p = 0.612). Comorbidities including diabetes, hypertension, chronic lung disease, and pneumonia were not found significantly different between COVID-19 and healthy subjects. However, the majority of COVID-19 subjects suffering from these diseases were in the mild group. The only important variable was the ischemic heart disease with a p-value of 0.041. Only 4 subjects were suffering from disease while having COVID-19, while no healthy subjects were recorded with this disease. All health conditions were found significantly different (p < 0.001) between COVID-19 and healthy subjects except for diarrhoea (p = 0.159). Significant health conditions included in this dataset were fever, cold, cough, muscle pain, loss of smell, sore throat, fatigue, and breathing difficulties. It is worth noting that only 4 subjects in the asymptomatic COVID-19 group were suffering from muscle pain, fatigue, and breathing difficulties. Analysis of MFCC Examples of the 13 MFCC extracted from the original shallow and deep breathing signals are illustrated in Figs 5 and 6, respectively, for COVID-19 and healthy subjects. Furthermore, the figures show MFCC values (after summing all coefficients) distributed as a normal distribution. From the figure, the normal distribution of COVID-19 subjects was slightly skewed to the right side of the mean, while the normal distribution of the healthy subjects was more towards the zero mean, indicating that it is better in representing a normal distribution. It is worth noting that the MFCC values of shallow breathing were lower than deep breathing in both COVID-19 and healthy subjects. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. Examples of the mel-frequency cepstral coefficients (MFCC) extracted from the shallow breathing dataset and illustrated as a normal distribution of summed coefficients. Showing: (a-c) COVID-19 subjects (asymptomatic, mild, moderate), (d-f) healthy subjects. https://doi.org/10.1371/journal.pone.0262448.g005 Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 6. Examples of the mel-frequency cepstral coefficients (MFCC) extracted from the deep breathing dataset and illustrated as a normal distribution of summed coefficients. Showing: (a-c) COVID-19 subjects (asymptomatic, mild, moderate), (d-f) healthy subjects. https://doi.org/10.1371/journal.pone.0262448.g006 Tables 2 and 3 show the values of the combined MFCC values, kurtosis, and skewness among all COVID-19 and healthy subjects (mean±std) for the shallow and deep breathing datasets, respectively. In both datasets, the kurtosis and skewness values for COVID-19 subjects were slightly higher than healthy subjects. Furthermore, the average combined MFCC values for COVID-19 were less than those for the healthy subjects. More specifically, in the shallow breathing dataset, a kurtosis and skewness of 4.65±15.97 and 0.59±1.74 was observed for COVID-19 subjects relative to 4.47±20.66 and 00.19±1.75 for healthy subjects. On the other hand, using the deep breathing dataset, COVID-19 subjects had a kurtosis and skewness of 20.82±152.99 and 0.65±4.35 compared to lower values of 3.23±6.06 and -0.36±1.08 for healthy subjects. In addition, a statstically significant difference (using the linear regression fitting algorithm) of <0.001 was obtained between COVID-19 and healthy subjects for the combined MFCC values of the shallow and deep breathing recordings. Furthermore, the skewness of the deep breathing recordings was found significantly different with a p-value of 0.014. It is worth noting that the skewness of the shallow breathing recordings was 0.095. Moreover, the kurtosis was not significant using both datasets’ recordings. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Normal distribution analysis (mean±std) of the combined mel-frequency cepstral coefficients (MFCCs) using the shallow breathing dataset. https://doi.org/10.1371/journal.pone.0262448.t002 Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Normal distribution analysis (mean±std) of the combined mel-frequency cepstral coefficients (MFCCs) using the deep breathing dataset. https://doi.org/10.1371/journal.pone.0262448.t003 Deep learning performance The overall performance of the proposed deep learning model is shown in Fig 7. From the figure, the model correctly predicted 113 and 114 COVID-19 and healthy subjects, respectively, using the shallow breathing dataset out of the 120 total subjects (Fig 7(a)). In addition, only 7 COVID-19 subjects were miss-classified as healthy, whereas only 6 subjects were wrongly classified as carrying COVID-19. The correct predictions number was slightly lower using the deep breathing dataset with a 109 and 112 for COVID-19 and healthy subjects, respectively. In addition, wrong predictions were also slightly higher with 11 COVID-19 and 8 healthy subjects. Therefore, the confusion matrices show percentages of proportion of 94.20% and 90.80% for COVID-19 subjects using the shallow and deep datasets, respectively. On the other hand, healthy subjects had percentages of 95.00% and 93.30% for both datasets, respectively. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 7. The performance of the deep learning model in predicting COVID-19 and healthy subjects using shallow and deep breathing datasets. Showing: (a) model’s predictions for both datasets and he corresponding confusion matrices, (b) evaluation metrics including accuracy, sensitivity, specificity, precision, and F1-score, (c) receiver operating characteristic (ROC) curves and corresponding area under the curve (AUROC) for COVID-19 and healthy subjects using both datasets. https://doi.org/10.1371/journal.pone.0262448.g007 The evaluation metrics (Fig 7(b)) calculated from these confusion matrices returned an accuracy measure of 94.58% and 92.08% for the shallow and deep datasets, respectively. Furthermore, the model had a sensitivity and specificity measures of 94.21%/94.96% for the shallow dataset and 93.16%/91.06% for the deep dataset. The precision was the highest measure obtained for the shallow dataset (95.00%), where as the deep dataset had the lowest value in the precision with a 90.83%. Lastly, the F1-score measures returned 94.61% and 91.98% for both datasets, respectively. To analyze the AUROC, Fig 7(c) shows the ROC curves of predictions using both the shallow and deep datasets. The shallow breathing dataset had an overall AUROC of 0.90 in predicting COVID-19 and healthy subjects, whereas the deep breathing dataset had a 0.86 AUROC, which is slightly lower performance in the prediction process. Additionally, the model had high accuracy measures in predicting asymptomatic COVID-19 subjects (Fig 8). Using the shallow breathing dataset, the model had a 100.00% accuracy by predicting all subjects correctly. On the other hand, using the deep breathing dataset, the model achieved an accuracy of 88.89% by missing two asymptomatic subjects. It is worth noting that few subjects had close scores (probabilities) to 0.5 using both datasets, however, the model correctly discriminated them from healthy subjects. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 8. Asymptomatic COVID-19 subjects’ predictions based on the proposed deep learning model. The model had a decision boundary of 0.5 to discriminate between COVID-19 and healthy subjects. The values represent a normalized probability regrading the confidence in predicting these subjects as carrying COVID-19. https://doi.org/10.1371/journal.pone.0262448.g008 Neural network activations Fig 9 shows the extracted neural network activations (deep-activated features) from the last layer (BiLSTM) for five examples from the COVID-19 and healthy subjects. These activations were obtained after applying the BiLSTM hidden units calculations on the flattened feature vector obtained from the CNN. The 512 hidden units are considered as the final deep-activated feature vector used to classify subjects into COVID-19 or healthy. By inspecting both COVID-19 (left column) and healthy (right column) subjects, it can be seen that the network learned successfully features that best maximize the margin between the two classes. For COVID-19, the activations were more spread all over the hidden units in a randomized manner, which could be due to the irregular breathing patterns seen in the original breathing sounds for COVID-19 subjects (Figs 2 and 3). On the other hand, healthy subjects had a close-to-regular patterns with higher power over the 60-128 and 200-350 hidden units. Similarly, this could be due to the normal breathing patterns observed in the healthy subjects breathing recordings. The ability of the neural network to acquire such differences in both classes suggest the potential of deep learning in the discrimination through 1D breathing sounds. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 9. Bi-directional long short-term memory (BiLSTM) network activations extracted for five examples from COVID-19 and healthy subjects using shallow breathing recordings. The activations were extracted from the last layer (applied on the flattened convolutional neural network (CNN) features vector) of the deep learning network (BiLSTM) for five examples from COVID-19 (a-f) and healthy (g-l) subjects. https://doi.org/10.1371/journal.pone.0262448.g009 Performance relative to current state-of-art To represent the performance of the proposed deep learning model relative to the current state-of-art studies, Table 4 shows the recent works on COVID-19 detection using machine learning and breathing/coughing recordings. The majority of studies have used coughing sounds to train deep learning networks. In addition, only two studies have utilized breathing sounds as input to the trained models [68, 69]. The only limitation in [69] is the heavy unbalance in favor of the normal subjects against COVID-19 subjects, which could have been the reason behind the high performance metrics achieved. In addition, authors in [68] use only 5 COVID-19 subjects, which does not ensure a generalized performance of deep learning networks. In contrary, the proposed study utilized a more balanced dataset with 120 COVID-19 subjects and the performance was higher than most of other studies. It is worth noting that most studies use web-based source for COVID-19 recordings, while in the proposed study breathing recordings were obtained from a smartphone app. In [69, 70], authors have used a smartphone app to acquire the recordings, however, they rely on coughing sounds, which makes it even more challenging to rely only on breathing sounds (as in the proposed study) and still achieve high performance. Additionally, the proposed study uses raw breathing signals (shallow and deep) to train deep learning models with the inclusion of best 20 features extracted from the raw signals and MFCC transformations, which was not the case in any study found in literature (majority require signal transformation to 2D images). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 4. Summary table of the current state-of-art works in COVID-19 detection using machine learning and breathing/coughing recordings. https://doi.org/10.1371/journal.pone.0262448.t004 Patient clinical information COVID-19 subjects included in this study had an average age of 34.04 years (± 13.45), while healthy subjects were slightly higher with an average of 36.02 years(±13.06). However, no significant difference was obtained for this variable across subjects (p = 0.149). It is worth noting that COVID-19 subjects with moderate conditions had a higher average age of 43.33 years. The distribution of male/female subjects across the two categories was close to 2:1 ratio with a majority of male subjects. Sex was not significantly different between COVID-19 and healthy subjects (p = 0.612). Comorbidities including diabetes, hypertension, chronic lung disease, and pneumonia were not found significantly different between COVID-19 and healthy subjects. However, the majority of COVID-19 subjects suffering from these diseases were in the mild group. The only important variable was the ischemic heart disease with a p-value of 0.041. Only 4 subjects were suffering from disease while having COVID-19, while no healthy subjects were recorded with this disease. All health conditions were found significantly different (p < 0.001) between COVID-19 and healthy subjects except for diarrhoea (p = 0.159). Significant health conditions included in this dataset were fever, cold, cough, muscle pain, loss of smell, sore throat, fatigue, and breathing difficulties. It is worth noting that only 4 subjects in the asymptomatic COVID-19 group were suffering from muscle pain, fatigue, and breathing difficulties. Analysis of MFCC Examples of the 13 MFCC extracted from the original shallow and deep breathing signals are illustrated in Figs 5 and 6, respectively, for COVID-19 and healthy subjects. Furthermore, the figures show MFCC values (after summing all coefficients) distributed as a normal distribution. From the figure, the normal distribution of COVID-19 subjects was slightly skewed to the right side of the mean, while the normal distribution of the healthy subjects was more towards the zero mean, indicating that it is better in representing a normal distribution. It is worth noting that the MFCC values of shallow breathing were lower than deep breathing in both COVID-19 and healthy subjects. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. Examples of the mel-frequency cepstral coefficients (MFCC) extracted from the shallow breathing dataset and illustrated as a normal distribution of summed coefficients. Showing: (a-c) COVID-19 subjects (asymptomatic, mild, moderate), (d-f) healthy subjects. https://doi.org/10.1371/journal.pone.0262448.g005 Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 6. Examples of the mel-frequency cepstral coefficients (MFCC) extracted from the deep breathing dataset and illustrated as a normal distribution of summed coefficients. Showing: (a-c) COVID-19 subjects (asymptomatic, mild, moderate), (d-f) healthy subjects. https://doi.org/10.1371/journal.pone.0262448.g006 Tables 2 and 3 show the values of the combined MFCC values, kurtosis, and skewness among all COVID-19 and healthy subjects (mean±std) for the shallow and deep breathing datasets, respectively. In both datasets, the kurtosis and skewness values for COVID-19 subjects were slightly higher than healthy subjects. Furthermore, the average combined MFCC values for COVID-19 were less than those for the healthy subjects. More specifically, in the shallow breathing dataset, a kurtosis and skewness of 4.65±15.97 and 0.59±1.74 was observed for COVID-19 subjects relative to 4.47±20.66 and 00.19±1.75 for healthy subjects. On the other hand, using the deep breathing dataset, COVID-19 subjects had a kurtosis and skewness of 20.82±152.99 and 0.65±4.35 compared to lower values of 3.23±6.06 and -0.36±1.08 for healthy subjects. In addition, a statstically significant difference (using the linear regression fitting algorithm) of <0.001 was obtained between COVID-19 and healthy subjects for the combined MFCC values of the shallow and deep breathing recordings. Furthermore, the skewness of the deep breathing recordings was found significantly different with a p-value of 0.014. It is worth noting that the skewness of the shallow breathing recordings was 0.095. Moreover, the kurtosis was not significant using both datasets’ recordings. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Normal distribution analysis (mean±std) of the combined mel-frequency cepstral coefficients (MFCCs) using the shallow breathing dataset. https://doi.org/10.1371/journal.pone.0262448.t002 Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Normal distribution analysis (mean±std) of the combined mel-frequency cepstral coefficients (MFCCs) using the deep breathing dataset. https://doi.org/10.1371/journal.pone.0262448.t003 Deep learning performance The overall performance of the proposed deep learning model is shown in Fig 7. From the figure, the model correctly predicted 113 and 114 COVID-19 and healthy subjects, respectively, using the shallow breathing dataset out of the 120 total subjects (Fig 7(a)). In addition, only 7 COVID-19 subjects were miss-classified as healthy, whereas only 6 subjects were wrongly classified as carrying COVID-19. The correct predictions number was slightly lower using the deep breathing dataset with a 109 and 112 for COVID-19 and healthy subjects, respectively. In addition, wrong predictions were also slightly higher with 11 COVID-19 and 8 healthy subjects. Therefore, the confusion matrices show percentages of proportion of 94.20% and 90.80% for COVID-19 subjects using the shallow and deep datasets, respectively. On the other hand, healthy subjects had percentages of 95.00% and 93.30% for both datasets, respectively. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 7. The performance of the deep learning model in predicting COVID-19 and healthy subjects using shallow and deep breathing datasets. Showing: (a) model’s predictions for both datasets and he corresponding confusion matrices, (b) evaluation metrics including accuracy, sensitivity, specificity, precision, and F1-score, (c) receiver operating characteristic (ROC) curves and corresponding area under the curve (AUROC) for COVID-19 and healthy subjects using both datasets. https://doi.org/10.1371/journal.pone.0262448.g007 The evaluation metrics (Fig 7(b)) calculated from these confusion matrices returned an accuracy measure of 94.58% and 92.08% for the shallow and deep datasets, respectively. Furthermore, the model had a sensitivity and specificity measures of 94.21%/94.96% for the shallow dataset and 93.16%/91.06% for the deep dataset. The precision was the highest measure obtained for the shallow dataset (95.00%), where as the deep dataset had the lowest value in the precision with a 90.83%. Lastly, the F1-score measures returned 94.61% and 91.98% for both datasets, respectively. To analyze the AUROC, Fig 7(c) shows the ROC curves of predictions using both the shallow and deep datasets. The shallow breathing dataset had an overall AUROC of 0.90 in predicting COVID-19 and healthy subjects, whereas the deep breathing dataset had a 0.86 AUROC, which is slightly lower performance in the prediction process. Additionally, the model had high accuracy measures in predicting asymptomatic COVID-19 subjects (Fig 8). Using the shallow breathing dataset, the model had a 100.00% accuracy by predicting all subjects correctly. On the other hand, using the deep breathing dataset, the model achieved an accuracy of 88.89% by missing two asymptomatic subjects. It is worth noting that few subjects had close scores (probabilities) to 0.5 using both datasets, however, the model correctly discriminated them from healthy subjects. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 8. Asymptomatic COVID-19 subjects’ predictions based on the proposed deep learning model. The model had a decision boundary of 0.5 to discriminate between COVID-19 and healthy subjects. The values represent a normalized probability regrading the confidence in predicting these subjects as carrying COVID-19. https://doi.org/10.1371/journal.pone.0262448.g008 Neural network activations Fig 9 shows the extracted neural network activations (deep-activated features) from the last layer (BiLSTM) for five examples from the COVID-19 and healthy subjects. These activations were obtained after applying the BiLSTM hidden units calculations on the flattened feature vector obtained from the CNN. The 512 hidden units are considered as the final deep-activated feature vector used to classify subjects into COVID-19 or healthy. By inspecting both COVID-19 (left column) and healthy (right column) subjects, it can be seen that the network learned successfully features that best maximize the margin between the two classes. For COVID-19, the activations were more spread all over the hidden units in a randomized manner, which could be due to the irregular breathing patterns seen in the original breathing sounds for COVID-19 subjects (Figs 2 and 3). On the other hand, healthy subjects had a close-to-regular patterns with higher power over the 60-128 and 200-350 hidden units. Similarly, this could be due to the normal breathing patterns observed in the healthy subjects breathing recordings. The ability of the neural network to acquire such differences in both classes suggest the potential of deep learning in the discrimination through 1D breathing sounds. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 9. Bi-directional long short-term memory (BiLSTM) network activations extracted for five examples from COVID-19 and healthy subjects using shallow breathing recordings. The activations were extracted from the last layer (applied on the flattened convolutional neural network (CNN) features vector) of the deep learning network (BiLSTM) for five examples from COVID-19 (a-f) and healthy (g-l) subjects. https://doi.org/10.1371/journal.pone.0262448.g009 Performance relative to current state-of-art To represent the performance of the proposed deep learning model relative to the current state-of-art studies, Table 4 shows the recent works on COVID-19 detection using machine learning and breathing/coughing recordings. The majority of studies have used coughing sounds to train deep learning networks. In addition, only two studies have utilized breathing sounds as input to the trained models [68, 69]. The only limitation in [69] is the heavy unbalance in favor of the normal subjects against COVID-19 subjects, which could have been the reason behind the high performance metrics achieved. In addition, authors in [68] use only 5 COVID-19 subjects, which does not ensure a generalized performance of deep learning networks. In contrary, the proposed study utilized a more balanced dataset with 120 COVID-19 subjects and the performance was higher than most of other studies. It is worth noting that most studies use web-based source for COVID-19 recordings, while in the proposed study breathing recordings were obtained from a smartphone app. In [69, 70], authors have used a smartphone app to acquire the recordings, however, they rely on coughing sounds, which makes it even more challenging to rely only on breathing sounds (as in the proposed study) and still achieve high performance. Additionally, the proposed study uses raw breathing signals (shallow and deep) to train deep learning models with the inclusion of best 20 features extracted from the raw signals and MFCC transformations, which was not the case in any study found in literature (majority require signal transformation to 2D images). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 4. Summary table of the current state-of-art works in COVID-19 detection using machine learning and breathing/coughing recordings. https://doi.org/10.1371/journal.pone.0262448.t004 Discussion This study demonstrated the importance of using deep learning for the detection of COVID-19 subjects, especially those who are asymptomatic. Furthermore, it elaborated on the significance of biological signals, such as breathing sounds, in acquiring useful information about the viral infection. Unlike the conventional lung auscultation techniques, i.e., electronic stethoscopes, to record breathing sounds, the study proposed herein utilized breathing sounds recorded via a smartphone microphone. The observations found in this study (highest accuracy: 94.58%) strongly suggest deep learning as a pre-screening tool for COVID-19 as well as an early detection technique prior to the gold standard RT-PCR assay. Smartphone-based breathing recordings Although current lung auscultation techniques provide high accuracy measures in detecting respiratory diseases [73–75], it requires subjects to be present at hospitals for equipment setup and testing preparation prior to data acquisition. Furthermore, it requires the availability of an experienced person, i.e., clinician or nurse, to take data from patients and store it in a database. Therefore, utilizing a smartphone device to acquire such data allows for a faster data acquisition process from subjects or patients while at the same time, provides highly comparable and acceptable diagnostic performance. In addition, smartphone-based lung auscultation ensures a better social distancing behaviour during lock downs due to pandemics such as COVID-19, thus, it allows for a rapid and time-efficient detection of diseases despite of strong restrictions. By visually inspecting COVID-19 and healthy subjects’ breathing recordings (Figs 2 and 3), an abnormal nature was usually observed by COVID-19 subjects, while healthy subjects had a more regular pattern during breathing. This could be related to the hidden characteristics of COVID-19 contaminated within lungs and exhibited during lung inhale and exhale [35, 38, 76]. Additionally, the MFCC transformation of COVID-19 and healthy subjects’ recordings returned similar observations. By quantitatively evaluating these coefficients when combined, COVID-19 subjects had a unique distribution (positively skewed) that can be easily distinguished from the one of healthy subjects. This gives an indication about the importance of further extracting the internal attributes carried not only by the recordings themselves, but rather by the additional MFC transformation of such recordings. Additionally, the asymptomatic subjects had a distribution of values that was close in shape to the distribution of healthy subjects, however, it was skewed towards the right side of the zero mean. This may be considered as a strong attribute when analyzing COVID-19 patients who do not exhibit any symptoms and thus, discriminating them easily from healthy subjects. Diagnosis of COVID-19 using deep learning It is essential to be able to gain the benefit of the recent advances in AI and computerized algorithms, especially during these hard times of COVID-19 spread worldwide. Deep learning not only provides high levels of performance, it also reduces the dependency on experts, i.e., clinicians and nurses, who are now suffering in handling the pandemic due to the huge and rapidly increasing number of infected patients [77–79]. Recently, the detection of COVID-19 using deep learning has reached high levels of accuracy through two-dimensional (2D) lung CT images [80–82]. Despite of such performance in discriminating and detecting COVID-19 subjects, CT imaging is considered high in cost and requires extra time to acquire testing data and results. Furthermore, it utilizes excessive amount of ionizing radiations (X-ray) that are usually harmful to the human body, especially for severely affected lungs. Therefore, the integration of biological sounds, as in breathing recordings, within a deep learning framework overcomes the aforementioned limitations, while at the same time provides acceptable levels of performance. The proposed deep learning framework had high levels of accuracy (94.58%) in discriminating between COVID-19 and healthy subjects. The structure of the framework was built to ensure a simple architecture, while at the same time to provide advanced features extraction and learning mechanisms. The combination between hand-crafted features and deep-activated features allowed for maximized performance capabilities within the model, as it learns through hidden and internal attributes as well as deep structural and temporal characteristics of recordings. The high sensitivity and specificity measures (94.21% and 94.96%, respectively) obtained in this study prove the efficiency of deep learning in distinguishing COVID-19 subjects (AUROC: 0.90). Additionally, it supports the field of deep learning research on the use of respiratory signals for COVID-19 diagnostics [39, 83]. Alongside the high performance levels, it was interesting to observe a 100.00% accuracy in predicting asymptomatic COVID-19 subjects. This could enhance the detection of this viral infection at a very early stage and thus, preventing it from developing to mild and moderate conditions or spreading to other people. Furthermore, this high performance levels were achieved through 1D signals instead of 2D images, which allowed the model to be simple and not memory exhausting. In addition, due to its simplicity and effective performance, it can be easily embedded within smartphone applications and internet-of-things tools to allow real-time and direct connectivity between the subject and family for care or healthcare authorities for services. Clinical relevance The statistical observations found in this study suggested that there is a significant difference between COVID-19 and healthy subjects for the ischemic heart disease comorbidity. This matches with the current discussions in literature about the correlation between COVID-19 and cardiac dysfunctionalities [84–86]. It was found in [84] that COVID-19 could induce myocardial injury, cardiac arrhythmia, and acute coronary syndrome. In addition, several health conditions related to the respiratory system were found significant in discriminating between COVID-19 and healthy subjects including fever, cold, and cough, which are the regular symptoms observed in most COVID-19 subjects. However, it was interesting to observe that muscle pain was significant, which matches with the previous WHO reports that states a percentage of 14.8% among COVID-19 subjects studied in China [87]. It is worth noting that diarrhoea was not significant in this study, which could show no correlation between COVID-19 and its existence in subjects. The utilization of smartphone-based breathing recordings within a deep learning framework may have the potential to provide a non-invasive, zero-cost, rapid pre-screening tool for COVID-19 in low-infected as well as servery-infected countries. Furthermore, it may be useful for countries who are not able of providing the RT-PCR test to everyone due to healthcare, economic, and political difficulties. Furthermore, instead of performing RT-PCR tests on daily or weekly basis, the proposed framework allows for easier, cost effective, and faster large-scale detection, especially for counties/areas who are putting high expenses on such tests due to logistical complications. Alongside the rapid nature of this approach, many healthcare service could be revived significantly by decreasing the demand on clinicians or nurses. In addition, due to the ability of successfully detecting asymptomatic subjects, it can decrease the need for extra equipment and costs associated with further medication after the development of the viral infection in patients. Clinically, it is better to have a faster connection between COVID-19 subjects and medical practitioners or health authorities to ensure continues monitoring for such cases and at the same time maintain successful contact tracing and social distancing. By embedding such approach within a smartphone applications or cloud-based networks, monitoring subjects, including those who are healthy or suspected to be carrying the virus, does not require the presence at clinics or testing points. Instead, it can be performed real-time through a direct connectivity with a medical practitioners. In addition, it can be completely done by the subject himself to self-test his condition prior to taking further steps towards the RT-PCR assay. Therefore, such approach could set an early alert to people, especially those who interacted with COVID-19 subjects or are asymptomatic, to go and further diagnose their case. Considering such mechanism in detecting COVID-19 could provide a better and well-organized approach that results in less demand for clinics and medical tests, and thus, enhances back the healthcare and economic sectors in various countries worldwide. Smartphone-based breathing recordings Although current lung auscultation techniques provide high accuracy measures in detecting respiratory diseases [73–75], it requires subjects to be present at hospitals for equipment setup and testing preparation prior to data acquisition. Furthermore, it requires the availability of an experienced person, i.e., clinician or nurse, to take data from patients and store it in a database. Therefore, utilizing a smartphone device to acquire such data allows for a faster data acquisition process from subjects or patients while at the same time, provides highly comparable and acceptable diagnostic performance. In addition, smartphone-based lung auscultation ensures a better social distancing behaviour during lock downs due to pandemics such as COVID-19, thus, it allows for a rapid and time-efficient detection of diseases despite of strong restrictions. By visually inspecting COVID-19 and healthy subjects’ breathing recordings (Figs 2 and 3), an abnormal nature was usually observed by COVID-19 subjects, while healthy subjects had a more regular pattern during breathing. This could be related to the hidden characteristics of COVID-19 contaminated within lungs and exhibited during lung inhale and exhale [35, 38, 76]. Additionally, the MFCC transformation of COVID-19 and healthy subjects’ recordings returned similar observations. By quantitatively evaluating these coefficients when combined, COVID-19 subjects had a unique distribution (positively skewed) that can be easily distinguished from the one of healthy subjects. This gives an indication about the importance of further extracting the internal attributes carried not only by the recordings themselves, but rather by the additional MFC transformation of such recordings. Additionally, the asymptomatic subjects had a distribution of values that was close in shape to the distribution of healthy subjects, however, it was skewed towards the right side of the zero mean. This may be considered as a strong attribute when analyzing COVID-19 patients who do not exhibit any symptoms and thus, discriminating them easily from healthy subjects. Diagnosis of COVID-19 using deep learning It is essential to be able to gain the benefit of the recent advances in AI and computerized algorithms, especially during these hard times of COVID-19 spread worldwide. Deep learning not only provides high levels of performance, it also reduces the dependency on experts, i.e., clinicians and nurses, who are now suffering in handling the pandemic due to the huge and rapidly increasing number of infected patients [77–79]. Recently, the detection of COVID-19 using deep learning has reached high levels of accuracy through two-dimensional (2D) lung CT images [80–82]. Despite of such performance in discriminating and detecting COVID-19 subjects, CT imaging is considered high in cost and requires extra time to acquire testing data and results. Furthermore, it utilizes excessive amount of ionizing radiations (X-ray) that are usually harmful to the human body, especially for severely affected lungs. Therefore, the integration of biological sounds, as in breathing recordings, within a deep learning framework overcomes the aforementioned limitations, while at the same time provides acceptable levels of performance. The proposed deep learning framework had high levels of accuracy (94.58%) in discriminating between COVID-19 and healthy subjects. The structure of the framework was built to ensure a simple architecture, while at the same time to provide advanced features extraction and learning mechanisms. The combination between hand-crafted features and deep-activated features allowed for maximized performance capabilities within the model, as it learns through hidden and internal attributes as well as deep structural and temporal characteristics of recordings. The high sensitivity and specificity measures (94.21% and 94.96%, respectively) obtained in this study prove the efficiency of deep learning in distinguishing COVID-19 subjects (AUROC: 0.90). Additionally, it supports the field of deep learning research on the use of respiratory signals for COVID-19 diagnostics [39, 83]. Alongside the high performance levels, it was interesting to observe a 100.00% accuracy in predicting asymptomatic COVID-19 subjects. This could enhance the detection of this viral infection at a very early stage and thus, preventing it from developing to mild and moderate conditions or spreading to other people. Furthermore, this high performance levels were achieved through 1D signals instead of 2D images, which allowed the model to be simple and not memory exhausting. In addition, due to its simplicity and effective performance, it can be easily embedded within smartphone applications and internet-of-things tools to allow real-time and direct connectivity between the subject and family for care or healthcare authorities for services. Clinical relevance The statistical observations found in this study suggested that there is a significant difference between COVID-19 and healthy subjects for the ischemic heart disease comorbidity. This matches with the current discussions in literature about the correlation between COVID-19 and cardiac dysfunctionalities [84–86]. It was found in [84] that COVID-19 could induce myocardial injury, cardiac arrhythmia, and acute coronary syndrome. In addition, several health conditions related to the respiratory system were found significant in discriminating between COVID-19 and healthy subjects including fever, cold, and cough, which are the regular symptoms observed in most COVID-19 subjects. However, it was interesting to observe that muscle pain was significant, which matches with the previous WHO reports that states a percentage of 14.8% among COVID-19 subjects studied in China [87]. It is worth noting that diarrhoea was not significant in this study, which could show no correlation between COVID-19 and its existence in subjects. The utilization of smartphone-based breathing recordings within a deep learning framework may have the potential to provide a non-invasive, zero-cost, rapid pre-screening tool for COVID-19 in low-infected as well as servery-infected countries. Furthermore, it may be useful for countries who are not able of providing the RT-PCR test to everyone due to healthcare, economic, and political difficulties. Furthermore, instead of performing RT-PCR tests on daily or weekly basis, the proposed framework allows for easier, cost effective, and faster large-scale detection, especially for counties/areas who are putting high expenses on such tests due to logistical complications. Alongside the rapid nature of this approach, many healthcare service could be revived significantly by decreasing the demand on clinicians or nurses. In addition, due to the ability of successfully detecting asymptomatic subjects, it can decrease the need for extra equipment and costs associated with further medication after the development of the viral infection in patients. Clinically, it is better to have a faster connection between COVID-19 subjects and medical practitioners or health authorities to ensure continues monitoring for such cases and at the same time maintain successful contact tracing and social distancing. By embedding such approach within a smartphone applications or cloud-based networks, monitoring subjects, including those who are healthy or suspected to be carrying the virus, does not require the presence at clinics or testing points. Instead, it can be performed real-time through a direct connectivity with a medical practitioners. In addition, it can be completely done by the subject himself to self-test his condition prior to taking further steps towards the RT-PCR assay. Therefore, such approach could set an early alert to people, especially those who interacted with COVID-19 subjects or are asymptomatic, to go and further diagnose their case. Considering such mechanism in detecting COVID-19 could provide a better and well-organized approach that results in less demand for clinics and medical tests, and thus, enhances back the healthcare and economic sectors in various countries worldwide. Conclusion This study suggests smartphone-based breathing sounds as a promising indicator for COVID-19 cases. It further recommends the utilization of deep learning as a pre-screening tool for such cases prior to the gold standard RT-PCR tests. The overall performance found in this study (accuracy 94.58%) in discriminating between COVID-19 and healthy subjects shows the potential of such approach. This study paves the way towards implementing deep learning in COVID-19 diagnostics by suggesting it as a rapid, time-efficient, and no-cost technique that does not violate social distancing restrictions during pandemics such as COVID-19. Acknowledgments The authors would like to acknowledge project Coswara and Dr. Sriram Ganapathy for their open-access database for COVID-19 breathing recordings.

Frenkel, Marie Ottilie;Pollak, Katja Mareike;Schilling, Oliver;Voigt, Laura;Fritzsching, Benedikt;Wrzus, Cornelia;Egger-Lampl, Sebastian;Merle, Uta;Weigand, Markus Alexander;Mohr, Stefan

doi: 10.1371/journal.pone.0261502pmid: 35041679

Background The COVID-19 pandemic has exerted great pressure on national health systems, which have aimed to ensure comprehensive healthcare at all times. Healthcare professionals working with COVID-19 patients are on the frontline and thereby confronted with enormous demands. Although early reports exist on the psychological impact of the pandemic on frontline medical staff working in Asia, little is known about its impact on healthcare professionals in other countries and across various work sectors. The present cross-sectional, online survey sought to investigate common work stressors among healthcare professionals, their psychological stress as well as coping resources during the pandemic. Methods A sample of 575 healthcare professionals (57% male) in three different sectors (hospital, prehospital emergency care, and outpatient service) reported their experiences concerning work and private stressors, psychological stress, and coping strategies between April 17, 2020 and June 5, 2020. To capture pandemic-specific answers, most of the items were adapted or newly developed. Exploratory factor analyses (EFA) were conducted to detect underlying latent factors relating to COVID-specific work stressors. In a next step, the effects of these latent stressors across various work sectors on psychological stress (perceived stress, fatigue, and mood) were examined by means of structural equation models (SEM). To add lived experience to the findings, responses to open-ended questions about healthcare professionals’ stressors, effective crisis measures and prevention, and individual coping strategies were coded inductively, and emergent themes were identified. Results The EFA revealed that the examined work stressors can be grouped into four latent factors: “fear of transmission”, “interference of workload with private life”, “uncertainty/lack of knowledge”, and “concerns about the team”. The SEM results showed that “interference of workload with private life” represented the pivotal predictor of psychological stress. “Concerns about the team” had stress-reducing effects. The latent stressors had an equal effect on psychological stress across work sectors. On average, psychological stress levels were moderate, yet differed significantly between sectors (all p < .001); the outpatient group experienced reduced calmness and more stress than the other two sectors, while the prehospital group reported lower fatigue than the other two sectors. The prehospital group reported significantly higher concerns about the team than the hospital group (p < .001). In their reports, healthcare professionals highlighted regulations such as social distancing and the use of compulsory masks, training, experience and knowledge exchange, and social support as effective coping strategies during the pandemic. The hospital group mainly mentioned organizational measures such as visiting bans as effective crisis measures, whereas the prehospital sector most frequently named governmental measures such as contact restrictions. Conclusion The study demonstrated the need for sector-specific crisis measures to effectively address the specific work stressors faced by the outpatient sector in particular. The results on pandemic-specific work stressors reveal that healthcare professionals might benefit from coping strategies that facilitate the utilization of social support. At the workplace, team commitment and knowledge exchange might buffer against adverse psychological stress responses. Schedules during pandemics should give healthcare workers the opportunity to interact with families and friends in ways that facilitate social support outside work. Future studies should investigate cross-sector stressors using a longitudinal design to identify both sector- and time-specific measures. Ultimately, an international comparison of stressors and measures in different sectors of healthcare systems is desirable. Introduction Globally, the COVID-19 pandemic has posed major challenges for public health systems. Besides stretching the capacities of intensive care units (ICU), healthcare professionals have represented the most critical resource for saving lives and limiting the impact of the pandemic [1]. As known from previous pandemic studies [2–4], healthcare professionals experience great psychological stress, while still being expected to act functionally at work. By acknowledging the need for constant healthcare provisions throughout the pandemic, effective crisis management that is targeted at reducing healthcare professionals’ psychological stress is required to protect their mental health, well-being, and functioning [5]. In view of this, the present study investigated COVID-19-specific work stressors, psychological stress, and coping resources among healthcare professionals, with the aim of improving mental health leaders’ and policy makers’ understanding of effective crisis management in pandemics. According to the transactional model of stress [6], a perceived discrepancy between the environmental demands placed on an individual and their available coping resources leads to stress. If environmental demands are perceived as threatening (i.e., stressor), an individual evaluates their available coping resources to determine whether they feel able to cope with the stressor. In times of heightened demands, such as during a pandemic, effective crisis prevention (taken before the stressful event), crisis measures (taken during the stressful event), as well as individual coping resources can be perceived as helpful to overcome the stressor, resulting in a "challenge" or neutral appraisal of the situation. However, if coping resources turn out to be inadequate, an individual experiences a negative psychological state of stress, which can include an increase of perceived stress and mood deteriorations [7]. Previous studies conducted during the outbreaks of SARS and Ebola have already shed some light on stressors that healthcare professionals have to face during pandemics. The major work stressors found in these studies included the feeling of risk towards getting infected and the fear of infecting one’s family, friends, and colleagues [8], worries arising due to uncertainty and stigmatization [8, 9] and a reluctance to work or contemplation of resignation [9]. Healthcare professionals reported high levels of work stress and reduced well-being [8–10], which were found to have long-term implications for their mental health [11–13]. Similar concerns have been raised about workload and its effects on the stress levels of healthcare professionals who are responsible for the treatment of patients with COVID-19 [1, 4, 14–18]. As a result, in recent months, several health organizations [19–21] (Inter-Agency Standing Committee, International Federation of Red Cross and Red Crescent Societies, World Health Organization) and research groups [4, 5, 14, 22, 23] have discussed a variety of COVID-19-specific interventions to support healthcare professionals’ ability to cope with their daily work. As a first step, the Inter-Agency Standing Committee provided a list of COVID-19-specific stressors of frontline healthcare professionals. Examples of items on this list were “strict bio-security measures”, “insufficient personal or energy capacity to implement basic self-care”, and “[the] fear that healthcare workers will pass COVID-19 onto their friends and family as a result of their work”. It is striking that, to date, almost all empirical studies on psychological stress during the ongoing pandemic focus on the effects of stress and fail to take a closer look at the stressors. Only one study integrated a selection of stressors at work [15], using an adapted short questionnaire on stressors during the SARS pandemic [24]. In summary, there is no comprehensive list of stressors formulated within current research that makes use of empirical data, as proposed by the Inter-Agency Standing Committee. Despite all the potential stressors, the availability of coping resources will ultimately determine whether the stressors lead to high stress levels [6]. While the risk of infection with COVID-19 is considered stressful in the general population [25] and among other frontline workers [1, 23, 26], healthcare professionals are well-trained in dealing with infectious patients, which might increase their perception of available coping resources [27–29]. Additional coping resources in the current pandemic might be governmental and organizational COVID-19 procedures, such as visiting bans in hospitals [30]. However, it is unclear which of these measures are perceived as effective in reducing psychological stress at work by healthcare professionals. This constitutes a further research gap addressed by this study. Due to the differential spread of COVID-19, countries have been affected more or less severely at different times. Thus, research focused on different countries in the beginning. A review of 14 empirical studies on the pandemic-related stress of healthcare professionals, published in the early stage of the COVID-19 pandemic from January to March 2020, revealed that the integrated studies only investigated individuals working in Asia, without extending attention to healthcare professionals’ work stressors during COVID-19 to the rest of the world [4]. National health systems, their preparedness for a pandemic as well as their capacities to deal with increased demands differ greatly between countries. For example, Germany was relatively well prepared due to early warnings from other countries and higher ICU capacities per capita. Additionally, the outpatient care sector through general practitioners and specialists in private practice represents an important pillar of the German healthcare system [4]. The outpatient care sector has emerged as a major factor for Germany’s strong enabling environment within the COVID-19 pandemic. However, previous studies focused on the comparison within different clinical departments and occupations in hospitals [4, 23]. Therefore, it remains unclear whether different sectors (i.e., hospital, prehospital emergency care, and outpatient sector) experience different stressors, perceive different coping resources to be effective, and are thus differently stressed. The overall aim of the present study was to identify target group-specific work stressors and coping resources to inform best practices in crisis management for further waves of COVID-19 or future outbreaks of pandemics. Building on existing assumptions of COVID-19 work-specific stressors [19], the study aimed to quantify common work stressors among healthcare professionals, assess their impact on professionals’ psychological stress and identify effective coping resources to counteract stressors by comparing hospital, prehospital emergency care and outpatient sectors. Methods The prospective repeated cross-sectional, observational study was conducted in Germany during the early stage of the COVID-19 pandemic from April 17, 2020 to June 5, 2020. The study was originally designed as a longitudinal survey to be administered at three different time points during the pandemic (April 17–24; May 8–15; May 29—June 5, 2020). However, the repeated recruitment of participants proved more difficult than expected due to low response rates in follow-up surveys. As a result, we treated the data as cross-sectional and do not report changes over time. In mid-March, the federal states [5] started to close daycare centres, kindergartens and schools. On March 23, 2020, the federal states and the national government [5] implemented a “contact restriction” measure, limiting public gatherings to two people (outside families), enforcing physical distance of at least five feet (1.5 meters), and closing many businesses. A gradual easing of physical distancing followed during our last measurement. The wearing of masks was only recommended in public during the first measurement and became obligatory during our second and third measurement. Sample and procedure The sample consisted of N = 575 healthcare professionals. To examine a large sample of healthcare staff from the hospital, prehospital emergency care and outpatient sectors, the survey was distributed online with the software SoSci Survey (http://www.soscisurvey.de). Participants were recruited by asking the medical directors of hospitals to forward the study invitation to staff and to post it on social media (e.g., medical forums, newspapers, Facebook groups, and other channels such as medical blogs and mailing lists). The Ethics Committee of the Faculty of Behavioral and Cultural Studies, Heidelberg University, Germany provided ethical approval for this study (approval number: AZ Fre 2020 1/1). Informed written consent was obtained from the participants. Participants received no financial compensation. Measures The survey length was kept as short as possible in order to ensure high participation rates and to minimize interference with professional duties. This was achieved by using a questionnaire that was originally designed for ecological momentary assessment [31], a list of COVID-19 related stressors for healthcare professionals [19] and several self-developed items, which were successfully used in previous studies [26, 32–34]. Explorative data was gained from six additional open-ended questions [see 26, S1 Table, column 2, 35], allowing participants to share their variety of ongoing experiences. On average, participants took 10–12 minutes to complete the questionnaire. Private stressors were assessed by asking for extraordinary private demands (e.g., whether one had caught COVID-19, death of a relative, ongoing divorce) [19]. Furthermore, participants were asked to rate the perceived stressfulness of 19 work stressors (e.g., “fear of getting infected”, “fear of infecting others”) on a scale ranging from 1 (not at all) to 7 (very). Potential stressors (see first column of Table 1) were adapted from a list of stressors for healthcare professionals in the early stages of the COVID-19 pandemic [19]. Participants could name additional work stressors in an open-ended question. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Means, standard deviations and bivariate correlations of different stressors. https://doi.org/10.1371/journal.pone.0261502.t001 Psychological stress. Actual perceived stress was measured using the single item “during the last week, I felt stressed out” which was rated from 1 (not at all) to 5 (very) [26, 32–34]. Fatigue was measured using the single item “during the last week, I felt fatigued” rated from 1 (not at all) to 5 (very) [26, 32–34]. Mood was measured by a six-item short version of the German Multidimensional Mood Questionnaire [31]. Three bipolar scales represent valence (V), energy (E) and calmness (C) [content–discontent (V–), tired–awake (E+), full of energy–without energy (E–), unwell–well (V+), agitated–calm (C+), relaxed–tense (C–)]. Each item was rated on a five-point scale ranging from 1 (not at all) to 5 (very). Wilhelm and Schoebi [31] reported good structural validity, sensitivity to change and reliability of this short scale. For the analyses, data from three items (i.e., V–, E–, C–) were reverse-coded. For V (α = .75), E (α = .79), and C (α = .74), average scores were calculated. Crisis management. After having indicated whether their area of work or function had changed through the COVID-19 pandemic, participants were asked to name effective crisis measures taken by the government, and/or the health system during the pandemic [see 26, S1 Table, second column]. In addition, participants were asked to rate their satisfaction with the measures on a scale ranging from 1 (not at all) to 7 (very). Furthermore, participants were asked in an open-ended question to name effective crisis prevention measures before the pandemic, i.e., those measures that have prepared them for the work demands during the current pandemic [26]. Again, participants were asked to rate how well their education and/or training prepared them for the current work demands during the COVID-19 crisis on a scale ranging from 1 (not at all) to 7 (very) [26]. Finally, participants were asked in an open-ended question to list their individual coping strategies. Data analysis Latent variable analyses. Exploratory factor analyses (EFA) were conducted to account for the intercorrelations between the 19 work stressors, examining the underlying latent factors of COVID-specific stressful experiences at work. In particular, we ran Maximum Likelihood EFA with Mplus 8.1, using the MLR estimator (to account for potential data non-normality) and modelling a “complex” data structure which accounts for dependencies of multiple observations from some respondents (for details see [36]). The Bayes Information Criterion (BIC) served as a decision criterium of the number of factors. We ran models with 1 to 9 factors and selected the solution which minimized the BIC [37]. Geomin rotation was used to allow for correlations between the factors. Finally, we included the solution obtained from the EFA in structural equation models (SEM) to examine the effects of the latent work stress factors on psychological stress. The chosen statistical method allows variability to be described among the observed correlated stressors in terms of a potentially lower number of latent (unobserved) variables called stress factors. Furthermore, we checked whether different work sectors affected latent work stress factors and psychological stress. We ran two SEM, as illustrated in Fig 1 (again using the MLR estimator and modelling a “complex” data structure). Notably, the four latent factors obtained from the EFA step were included in these SEM by keeping the 4×19 (unstandardized) factor loadings, as well as the 19 work stressor intercepts estimated for the EFA solution, fixed in all these analyses. First, we modelled the observed scores of five psychological stress outcomes (i.e., perceived stress, fatigue, energy, valence, and calmness) predicted by the latent work stress factors (M1). Second, we analyzed work-sector-related differences by means of a multigroup SEM (M2) and Wald tests of parameter constraints [36], assessing whether the path coefficients of different sectors could be equated without significant loss of model fit. We also ran both models controlling for age and sex i.e., both were added as predictors for the outcome, which did not result in any notable changes in the results reported below (regarding neither values nor the significance of the estimated coefficients). Therefore, for reasons of space, we present the results of the “uncontrolled” analyses below. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Structural equation modeling: Analysis plan. F = factor, F1 = fear of transmission, F2 = interference of workload with private life, F3 = uncertainty/lack of knowledge, F4 = concerns about the team. M1 = Overall Model, M2 = Work Sector. https://doi.org/10.1371/journal.pone.0261502.g001 Qualitative analyses. The open-ended questions were analyzed and coded on the basis of the deductive category assignment in the qualitative content analysis according to Mayring [38]. Quotations were derived from the answers to the open-ended questions and served as data units in the analysis. Each data unit consists of an independently interpretable and meaningful unit. When a participant’s quotation addressed more than one meaningful issue, it was divided into multiple data units. For example, when a respondent listed three additional work demands, this answer was split into three data units, each listing one work demand. For the coding system, main categories and subcategories were derived inductively for each question based on the material from approximately 50% of the dataset. All data units were assigned to the existing main categories. If a data unit could not be clearly assigned to any of the defined subcategories, it was sorted into a main category. These frequencies were used to identify main themes, changes or differences between sectors, which are in turn described in more detail by providing individual quotations. We reported an overall response rate for each open-ended question, which refers to the percentage of surveys that have covered the respective open-ended question. It should be noted that, due to the repeated participation of few participants, the overall response rate might differ slightly from the percentage of participants that have answered the respective question. Sample and procedure The sample consisted of N = 575 healthcare professionals. To examine a large sample of healthcare staff from the hospital, prehospital emergency care and outpatient sectors, the survey was distributed online with the software SoSci Survey (http://www.soscisurvey.de). Participants were recruited by asking the medical directors of hospitals to forward the study invitation to staff and to post it on social media (e.g., medical forums, newspapers, Facebook groups, and other channels such as medical blogs and mailing lists). The Ethics Committee of the Faculty of Behavioral and Cultural Studies, Heidelberg University, Germany provided ethical approval for this study (approval number: AZ Fre 2020 1/1). Informed written consent was obtained from the participants. Participants received no financial compensation. Measures The survey length was kept as short as possible in order to ensure high participation rates and to minimize interference with professional duties. This was achieved by using a questionnaire that was originally designed for ecological momentary assessment [31], a list of COVID-19 related stressors for healthcare professionals [19] and several self-developed items, which were successfully used in previous studies [26, 32–34]. Explorative data was gained from six additional open-ended questions [see 26, S1 Table, column 2, 35], allowing participants to share their variety of ongoing experiences. On average, participants took 10–12 minutes to complete the questionnaire. Private stressors were assessed by asking for extraordinary private demands (e.g., whether one had caught COVID-19, death of a relative, ongoing divorce) [19]. Furthermore, participants were asked to rate the perceived stressfulness of 19 work stressors (e.g., “fear of getting infected”, “fear of infecting others”) on a scale ranging from 1 (not at all) to 7 (very). Potential stressors (see first column of Table 1) were adapted from a list of stressors for healthcare professionals in the early stages of the COVID-19 pandemic [19]. Participants could name additional work stressors in an open-ended question. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Means, standard deviations and bivariate correlations of different stressors. https://doi.org/10.1371/journal.pone.0261502.t001 Psychological stress. Actual perceived stress was measured using the single item “during the last week, I felt stressed out” which was rated from 1 (not at all) to 5 (very) [26, 32–34]. Fatigue was measured using the single item “during the last week, I felt fatigued” rated from 1 (not at all) to 5 (very) [26, 32–34]. Mood was measured by a six-item short version of the German Multidimensional Mood Questionnaire [31]. Three bipolar scales represent valence (V), energy (E) and calmness (C) [content–discontent (V–), tired–awake (E+), full of energy–without energy (E–), unwell–well (V+), agitated–calm (C+), relaxed–tense (C–)]. Each item was rated on a five-point scale ranging from 1 (not at all) to 5 (very). Wilhelm and Schoebi [31] reported good structural validity, sensitivity to change and reliability of this short scale. For the analyses, data from three items (i.e., V–, E–, C–) were reverse-coded. For V (α = .75), E (α = .79), and C (α = .74), average scores were calculated. Crisis management. After having indicated whether their area of work or function had changed through the COVID-19 pandemic, participants were asked to name effective crisis measures taken by the government, and/or the health system during the pandemic [see 26, S1 Table, second column]. In addition, participants were asked to rate their satisfaction with the measures on a scale ranging from 1 (not at all) to 7 (very). Furthermore, participants were asked in an open-ended question to name effective crisis prevention measures before the pandemic, i.e., those measures that have prepared them for the work demands during the current pandemic [26]. Again, participants were asked to rate how well their education and/or training prepared them for the current work demands during the COVID-19 crisis on a scale ranging from 1 (not at all) to 7 (very) [26]. Finally, participants were asked in an open-ended question to list their individual coping strategies. Psychological stress. Actual perceived stress was measured using the single item “during the last week, I felt stressed out” which was rated from 1 (not at all) to 5 (very) [26, 32–34]. Fatigue was measured using the single item “during the last week, I felt fatigued” rated from 1 (not at all) to 5 (very) [26, 32–34]. Mood was measured by a six-item short version of the German Multidimensional Mood Questionnaire [31]. Three bipolar scales represent valence (V), energy (E) and calmness (C) [content–discontent (V–), tired–awake (E+), full of energy–without energy (E–), unwell–well (V+), agitated–calm (C+), relaxed–tense (C–)]. Each item was rated on a five-point scale ranging from 1 (not at all) to 5 (very). Wilhelm and Schoebi [31] reported good structural validity, sensitivity to change and reliability of this short scale. For the analyses, data from three items (i.e., V–, E–, C–) were reverse-coded. For V (α = .75), E (α = .79), and C (α = .74), average scores were calculated. Crisis management. After having indicated whether their area of work or function had changed through the COVID-19 pandemic, participants were asked to name effective crisis measures taken by the government, and/or the health system during the pandemic [see 26, S1 Table, second column]. In addition, participants were asked to rate their satisfaction with the measures on a scale ranging from 1 (not at all) to 7 (very). Furthermore, participants were asked in an open-ended question to name effective crisis prevention measures before the pandemic, i.e., those measures that have prepared them for the work demands during the current pandemic [26]. Again, participants were asked to rate how well their education and/or training prepared them for the current work demands during the COVID-19 crisis on a scale ranging from 1 (not at all) to 7 (very) [26]. Finally, participants were asked in an open-ended question to list their individual coping strategies. Data analysis Latent variable analyses. Exploratory factor analyses (EFA) were conducted to account for the intercorrelations between the 19 work stressors, examining the underlying latent factors of COVID-specific stressful experiences at work. In particular, we ran Maximum Likelihood EFA with Mplus 8.1, using the MLR estimator (to account for potential data non-normality) and modelling a “complex” data structure which accounts for dependencies of multiple observations from some respondents (for details see [36]). The Bayes Information Criterion (BIC) served as a decision criterium of the number of factors. We ran models with 1 to 9 factors and selected the solution which minimized the BIC [37]. Geomin rotation was used to allow for correlations between the factors. Finally, we included the solution obtained from the EFA in structural equation models (SEM) to examine the effects of the latent work stress factors on psychological stress. The chosen statistical method allows variability to be described among the observed correlated stressors in terms of a potentially lower number of latent (unobserved) variables called stress factors. Furthermore, we checked whether different work sectors affected latent work stress factors and psychological stress. We ran two SEM, as illustrated in Fig 1 (again using the MLR estimator and modelling a “complex” data structure). Notably, the four latent factors obtained from the EFA step were included in these SEM by keeping the 4×19 (unstandardized) factor loadings, as well as the 19 work stressor intercepts estimated for the EFA solution, fixed in all these analyses. First, we modelled the observed scores of five psychological stress outcomes (i.e., perceived stress, fatigue, energy, valence, and calmness) predicted by the latent work stress factors (M1). Second, we analyzed work-sector-related differences by means of a multigroup SEM (M2) and Wald tests of parameter constraints [36], assessing whether the path coefficients of different sectors could be equated without significant loss of model fit. We also ran both models controlling for age and sex i.e., both were added as predictors for the outcome, which did not result in any notable changes in the results reported below (regarding neither values nor the significance of the estimated coefficients). Therefore, for reasons of space, we present the results of the “uncontrolled” analyses below. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Structural equation modeling: Analysis plan. F = factor, F1 = fear of transmission, F2 = interference of workload with private life, F3 = uncertainty/lack of knowledge, F4 = concerns about the team. M1 = Overall Model, M2 = Work Sector. https://doi.org/10.1371/journal.pone.0261502.g001 Qualitative analyses. The open-ended questions were analyzed and coded on the basis of the deductive category assignment in the qualitative content analysis according to Mayring [38]. Quotations were derived from the answers to the open-ended questions and served as data units in the analysis. Each data unit consists of an independently interpretable and meaningful unit. When a participant’s quotation addressed more than one meaningful issue, it was divided into multiple data units. For example, when a respondent listed three additional work demands, this answer was split into three data units, each listing one work demand. For the coding system, main categories and subcategories were derived inductively for each question based on the material from approximately 50% of the dataset. All data units were assigned to the existing main categories. If a data unit could not be clearly assigned to any of the defined subcategories, it was sorted into a main category. These frequencies were used to identify main themes, changes or differences between sectors, which are in turn described in more detail by providing individual quotations. We reported an overall response rate for each open-ended question, which refers to the percentage of surveys that have covered the respective open-ended question. It should be noted that, due to the repeated participation of few participants, the overall response rate might differ slightly from the percentage of participants that have answered the respective question. Latent variable analyses. Exploratory factor analyses (EFA) were conducted to account for the intercorrelations between the 19 work stressors, examining the underlying latent factors of COVID-specific stressful experiences at work. In particular, we ran Maximum Likelihood EFA with Mplus 8.1, using the MLR estimator (to account for potential data non-normality) and modelling a “complex” data structure which accounts for dependencies of multiple observations from some respondents (for details see [36]). The Bayes Information Criterion (BIC) served as a decision criterium of the number of factors. We ran models with 1 to 9 factors and selected the solution which minimized the BIC [37]. Geomin rotation was used to allow for correlations between the factors. Finally, we included the solution obtained from the EFA in structural equation models (SEM) to examine the effects of the latent work stress factors on psychological stress. The chosen statistical method allows variability to be described among the observed correlated stressors in terms of a potentially lower number of latent (unobserved) variables called stress factors. Furthermore, we checked whether different work sectors affected latent work stress factors and psychological stress. We ran two SEM, as illustrated in Fig 1 (again using the MLR estimator and modelling a “complex” data structure). Notably, the four latent factors obtained from the EFA step were included in these SEM by keeping the 4×19 (unstandardized) factor loadings, as well as the 19 work stressor intercepts estimated for the EFA solution, fixed in all these analyses. First, we modelled the observed scores of five psychological stress outcomes (i.e., perceived stress, fatigue, energy, valence, and calmness) predicted by the latent work stress factors (M1). Second, we analyzed work-sector-related differences by means of a multigroup SEM (M2) and Wald tests of parameter constraints [36], assessing whether the path coefficients of different sectors could be equated without significant loss of model fit. We also ran both models controlling for age and sex i.e., both were added as predictors for the outcome, which did not result in any notable changes in the results reported below (regarding neither values nor the significance of the estimated coefficients). Therefore, for reasons of space, we present the results of the “uncontrolled” analyses below. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Structural equation modeling: Analysis plan. F = factor, F1 = fear of transmission, F2 = interference of workload with private life, F3 = uncertainty/lack of knowledge, F4 = concerns about the team. M1 = Overall Model, M2 = Work Sector. https://doi.org/10.1371/journal.pone.0261502.g001 Qualitative analyses. The open-ended questions were analyzed and coded on the basis of the deductive category assignment in the qualitative content analysis according to Mayring [38]. Quotations were derived from the answers to the open-ended questions and served as data units in the analysis. Each data unit consists of an independently interpretable and meaningful unit. When a participant’s quotation addressed more than one meaningful issue, it was divided into multiple data units. For example, when a respondent listed three additional work demands, this answer was split into three data units, each listing one work demand. For the coding system, main categories and subcategories were derived inductively for each question based on the material from approximately 50% of the dataset. All data units were assigned to the existing main categories. If a data unit could not be clearly assigned to any of the defined subcategories, it was sorted into a main category. These frequencies were used to identify main themes, changes or differences between sectors, which are in turn described in more detail by providing individual quotations. We reported an overall response rate for each open-ended question, which refers to the percentage of surveys that have covered the respective open-ended question. It should be noted that, due to the repeated participation of few participants, the overall response rate might differ slightly from the percentage of participants that have answered the respective question. Results Sample description A total of N = 615 observations were used for the analyses. Notably, these observations were obtained from 575 participants (57% male), including eight individuals (1.4%) who participated at all three measurement points and 24 individuals (4.2%) who participated twice. Thus, a total of 543 (94.4%) persons participated only at one of three measurement points, thereby delivering no “truly longitudinal” information on intra-individual change. The participants’ age ranged from <20 to 65–69 years. More than half of the participants (58.4%) were aged between 25 and 44 years. Participants’ work experience ranged from 0 to more than 40 years. In all sectors, roughly half of the participants reported a change of their work area or function due to the pandemic (55% in the hospital sector, 44% in the prehospital sector, and 48% in the outpatient sector). The subsample from the hospital sector (n = 254) included medical doctors, nurses, and undergraduate medical students. The prehospital subsample (n = 250) comprised emergency physicians and paramedics. Regarding the outpatient sector (n = 108), participants were medical doctors, undergraduate medical students, and medical assistants working at general, family or pediatric practices which specialized in the treatment of COVID-19 patients. Three participants did not provide any information about their work sector and were thus excluded from the analysis which focused on differences between work sectors. Private stressors during the pandemic Most of the healthcare professionals did not report private stressors (overall response rate of 29.3%). One third of the given answers was related to the main subject “worries about the health of relatives”. “Caregiving duties” was the second most frequently mentioned theme (S3 Table). In relation to the first main subject (worries about relatives), healthcare professionals mostly mentioned not only relatives and/or friends who belonged to certain risk groups but also stigmatization from family members and friends who were afraid of getting infected. A few participants reported extreme accumulation of family duties or critical life events, such as the dead of a relative or friend. – Wife is afraid of being infected by me. Constant domestic discussions about the dangerous nature of the situation. (male, 65–69 years old, prehospital sector) In relation to the second main subject (caregiving duties), participants most frequently mentioned childcare and home schooling as stressful caregiving duties. – Childcare with 2 full-time working parents with very limited home office facilities. (female, 40–44 years old, prehospital sector) Work stressors during the pandemic Regarding work-related stressors, participants’ ratings on the perceived stressfulness of the 19 pandemic-specific stressors (Table 1) revealed that the necessity of having “strict safety measures” was considered as most stressful. “Fear of getting infected” and “feeling of being isolated from usual work team” were comparably low work stressors. To detect groups of stressors that belong together or co-occurred frequently, we conducted an exploratory factor analyses (EFA) of the 19 distinct stressors. Intercorrelations between the 19 different stressors are presented in Table 1. The EFA revealed four distinct factors (i.e., groups of stressors) because the 4-factorial model demonstrated the best fit with the lowest information criterion BIC (BICs for models with 1 to 9 factors, respectively: 45040.5, 44482.4, 44303.5, 44280.3, 44287.8, 44309.2, 44330.0, 44367.1, 44406.6). In terms of widely used fit indices, this 4-factor model provided good to acceptable fit to the data (RMSEA = 0.06, CFI = 0.947, SRMR = 0.027). Hence, four factors were extracted in total (S1 Table for factor correlations, and S2 Table for factor loadings). In interpreting the core items with highest loadings/factor correlations for each factor (S2 Table), the first factor can be characterized as “fear of transmission” (core items: “fear of infecting others”, λ = 0.898; “fear of infecting friends and family”, λ = 0.881), the second as “interference of workload with private life” (core items: “reduced capacity to use social support due to long work time and stigmatization of healthcare workers”, λ = 0.826; “insufficient capacity to implement basic self-care due to lack of time and energy”, λ = 0.734), and the third as “uncertainty/lack of knowledge” (core items: “information overload of constantly changing information”, λ = 0.740; “no clear instructions”, λ = 0.909). The significance of the fourth factor seems less clear. Whereas even the items with relatively high loadings on this factor show higher loadings on one of the other factors, this fourth factor revealed the lowest correlations with the others. This suggests that this fourth factor reflects some additional source of variance which is largely independent of the other factors, which affects some items that mainly “belong” to one of the first three factors. Notably, the wording of all items loading relatively high on this factor have in common a notion of working within a team (e.g., highest loading core items: “feeling isolated due to separation from usual work team”, λ = 0.339; “stigmatization of oneself and other healthcare personnel working with COVID-19 infected patients”, λ = 0.311). The factor comprises stressors that are likely to affect the respondent more, the more they feel integrated into the work team. The work teams consist of colleagues in the healthcare sector in which the respondents work. Therefore, this factor may be interpreted in terms of “concerns about the team”, a factor which may include both additional stress in terms of responsibility for the team and resilience due to increased team integration. Additional work stressors which were asked about through an open-ended question were mentioned in only 20% of the surveys. “Non-compliance of the society” (12%), “medical supply shortage” (11%) and “increased workload” (11%) were mentioned most frequently. Psychological stress The results showed that “interference of workload with private life” was the pivotal predictor of psychological stress, whereas “concerns about the team” had stress-reducing effects. Generally, stressors had equal effects on psychological stress across work sectors. We now present the results in detail. Results from the structural equation models M1 and M2 (estimates of intercepts and path coefficients) are summarized in Table 2 (see Fig 1 for an illustration of the models) below. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Structural equation model results: estimates of factor means, intercepts, and path coefficients from models 1 and 2. https://doi.org/10.1371/journal.pone.0261502.t002 Overall Model (M1): The factor “fear of transmission” did not reveal any significant effects predicting outcomes of energy, valence, calmness, stress, and fatigue (all ps > .09). The factor "uncertainty/lack of knowledge" showed only small effects predicting these outcomes, with absolute coefficient values ranging from 0.07 to 0.15 (although it was significant at p < .05 in predicting valence and calmness). The factor “interference of workload with private life” was found to be the pivotal factor in predicting low energy (b = -0.51, p < .001), valence (b = -0.38, p < .001), calmness (b = -0.44, p < .001), and high levels of stress (b = 0.53, p < .001), and fatigue (b = 0.51, p < .001). In contrast, the factor “concerns about the team” showed some smaller effects, predicting higher energy (b = 0.219, p < .001) and calmness (b = 0.161, p < .01) and less stress (b = -0.153, p < .01) and fatigue (b = -0.190, p < .001). M1 fitted the data well (RMSEA = 0.035, CFI = 0.967, SRMR = 0.057). Work Sector (M2): The overall test for the equality of all coefficients across the three sectors did not show a significant result (W[40] = 50.99, p = .11), indicating that between-sector differences in the path coefficients could be constrained equally without loss of model fit. Thus, the effects of the identified factors on the outcomes did not differ substantially between the work sectors. Regarding the factor means, the factor “concerns about the team” was significantly higher in the prehospital sector than in the hospital sector. There were no other between-sector mean differences. Notably, however, the outcome intercepts differed between the sectors (overall Wald test: W[10] = 23.22, p = .01). Further tests on each of the outcomes showed that these between-sector differences are significant for calmness (W[2] = 15.01, p < .001), stress (W[2] = 10.30, p = .006) and fatigue (W[2] = 14.37, p < .001), and without α-adjustment for multiple testing also for valence (W[2] = 6.34, p = .04). Thus, as intercept differences could be taken as an indicator of direct sector effects on the outcomes (i.e., effects which are unmediated by the stressor factors), respondents from the outpatient group experienced reduced calmness and felt more stressed, whereas the prehospital group reported lower fatigue. M2 fitted slightly worse than M1, but still fairly well (RMSEA = 0.047, CFI = 0.942, SRMR = 0.065). Crisis management Regarding crisis management, healthcare professionals highlighted social distancing and compulsory masks, training, experience and knowledge exchange, and social support as effective coping strategies during the pandemic. The hospital group mainly mentioned organizational measures such as visiting bans as effective crisis measures, whereas the prehospital sector most frequently named governmental measures such as contact restrictions. We will now proceed to describe the results in detail. Descriptive data of the satisfaction ratings for crisis measures and satisfaction with crisis prevention are presented in Table 3. Satisfaction with crisis measures and prevention was rated medium across all sectors on average, with means ranging from 4.23 to 4.54 for crisis measures and from 3.64 to 3.71 for crisis prevention. Main categories of themes derived from the open-ended questions and frequencies of quotations are reported for each sector (S3 Table). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Satisfaction with short-term crisis measures and crisis prevention. https://doi.org/10.1371/journal.pone.0261502.t003 Effective crisis measures. More than half of the healthcare professionals responded to the open-ended question about subjectively perceived effective crisis measures (57.2%). On the governmental level, healthcare professionals mainly perceived social distancing regulations and compulsory oronasal masks for the general population as effective. – Requirements to keep distance to others create awareness of the gravity of the situation. (male, 50–54, hospital sector) – Patients are used to masks, which they now must also wear in ambulances and hospitals in case of emergency. (male, 30–34 years old, prehospital sector) On the organizational level, healthcare professionals from all sectors perceived visiting bans, bans on elective surgery, and supply of protective equipment for healthcare staff as useful. – Visiting bans of relatives in hospitals; much quieter surroundings and less drama (male, 20–24 years old, prehospital sector) – No plannable surgeries, therefore [there is] more staff on [the] intensive care unit. (female, 45–49 years old, hospital sector) It appears that the perceived effectiveness of these regulations varied across sectors. Organizational measures were mostly mentioned in the hospital sector, whereas governmental measures were more often introduced in the prehospital sector. At the same time, approximately a fifth of the given answers were subsumed under the category “no effective crisis measures have been taken”. In the descriptions of the healthcare professionals, different reasons can be identified for this outcome: a) no measures were taken because no measures were necessary, b) measures should have been taken, but they were not, c) measures were taken but were not sufficient or effective, and d) measures that were taken even aggravated the work. a) We have been relatively well prepared by the annual influenza wave. (female, 30–34 years old, outpatient sector) b) Nothing at all. In my opinion the supply of protective masks and gowns was insufficient. (male, 20–24 years old, hospital sector) c) Since unfortunately people are very inconsistent with the “rule”, which is not even controlled properly, I am very angry. (male, 20–24 years old, sector unknown) d) None! I feel no relief from the crisis measures taken. On the contrary, my field of activity has expanded. (male, 55–59 years old, hospital sector) Effective crisis prevention. Half of the healthcare professionals named effective crisis prevention (overall response rate of 51.4%). Healthcare professionals perceived three main factors as effective in crisis prevention (which together made up more than 67% of the answers, S3 Table): (1) aspects of their professional training, (2) general work experience and exchange, and (3) crisis measures implemented at an early stage. Regarding the training of healthcare professionals, participants mostly mentioned general hygienic training (infection protection courses dealing with highly infectious patients), followed by specific hygienic training related to SARS CoV-2 and further vocational training with a focus on crisis management. In particular, they described repetition and regular updating of such courses as key principles that lead to additional safety when dealing with infectious patients. – Re-training on materials and the correct way to put on and take off the infection protection equipment. (female, 20–24 years old, prehospital sector) – A bunch of good training courses held by the employer of the clinic, e.g., a course related to protective clothing as well as additional information via e-mails I received regularly. (female, 60–64 years old, hospital sector) Besides training, healthcare professionals reported work experience and sharing current work experiences with colleagues as an effective resource during the pandemic. Having experience in dealing with highly infectious patients was perceived as helpful. Occasionally, participants reported that experiences and lessons learnt from other viruses (e.g., avian influenza), other pandemics (e.g., swine flu pandemic) or similar contexts (civil protection activities) prepared them for the COVID-19 pandemic. Sharing current work experience with colleagues through active exchange was perceived as effective and seemed to foster team commitment. – Extensive contact with isolated patients even before the pandemic—routine prevents fear! (male, 35–39 years old, prehospital sector) – Teaching units and good exchange among each other during my currently ongoing professional training. (male, 30–34 years old, hospital sector) – Link with Italian colleagues to exchange expertise. (male, 40–44 years old, hospital sector) – We had regular team meetings; our superior kept us up to date with all the news. (female, 50–54 years old, outpatient sector) One fifth of the reports described early-stage crisis measures as effective in helping crisis prevention. Specifically, the flow of relevant information through internal channels of information including emails or intranet was often mentioned. Nationwide daily information channels such as the Robert Koch Institute (RKI, German Federal Authority for Infectious Diseases) or COVID-19-specific podcasts were also named. Furthermore, the formulation of clear guidelines and recommendations for action and the timely provision of protective material were both perceived as helpful. – Information by my superior, information from the clinic management, but also passing on information among the assistants/specialists, information from senior nurses. (female, 40–44 years old, hospital sector) – Daily "employee news" from the management to the employees in the form of e-mail and information paper at the beginning of the shift. (male, 30–34 years old, hospital sector) – Clear specification of hygiene management in the clinic significantly improved interdisciplinary communication in the clinic and in the department, daily updates by the crisis management team of the clinic and the department also by e-mail to my private e-mail address. – Many visual representations of COVID and hygiene procedures by RKI, but also on the internet by professional associations or Free Open Access Medical Education.  (male, 60–64 years old, hospital sector) – Availability of significantly more protective material! Protective measures before and after each patient contact. (male, 20–24 years old, prehospital sector) It appears that the perceived effectiveness of crisis prevention varied across sectors. General work experience and exchange were mostly mentioned in the hospital and outpatient sectors, whereas early-stage crisis measures were more often applied in the prehospital sector. One seventh of the answers were assigned to the category “no effective crisis prevention has been taken” (see S3 Table). Analogous to the acute crisis measures, different reasons for this answer can be derived from the reports: a) preventive measures should have been taken, but they were not, b) no crisis prevention was possible because the situation was unexpected and dynamic, and, c) the measures taken were insufficient. Irrespective of the judgment of how prepared they felt, some healthcare professionals specified that there was d) no specific crisis management or pandemic-specific training. Additionally, some participants perceived no crisis prevention measures on an organizational level but emphasized the use of individual strategies. a) None. […] Bad (as well as fake) videos of how to put on protective clothing. (male, 25–29 years old, hospital sector) b) None at all. Nobody knew how to deal with trainees or how to deal with the training and further education for rescue service personnel. (female, 35–39 years old, prehospital sector) c) None, nobody knew about the Coronavirus before! (male, 35–39 years old, hospital sector) d) None. Common sense! (male, 25–39 years old, hospital sector) Individual coping. In the open-ended question about individual coping strategies (which had an overall response rate of 83.4%), three areas were frequently mentioned (which together made up 80% of the given answers, S3 Table): (1) social support, (2) hobbies/leisure activities, and (3) mental strategies. Within social contacts, the family is perceived as mainly supportive, while friends and colleagues were also mentioned as important. Most of the named hobbies and leisure activities were sport activities. Walks and opportunities to enjoy nature were also frequently reported. Regarding mental strategies, most participants reported the benefits of distractive activities such as watching videos, listening to audio books, social media, or online shopping. In contrast, other respondents mentioned a deliberately reduced media consumption, although these reports were less frequently noted. The use of relaxation/meditation techniques was often reported as a distraction. Furthermore, a few participants answered with thoughts or plans to quit their job, while others mentioned altruistic or intrinsic motives at work. – I don’t enjoy working for the most part for the first time in my life. (male, 35–39 years old, outpatient sector) – I think about changing my job more often. (male, 40–44 years old, prehospital sector) – The desire to help others with it. (female, 25–29 years old, outpatient sector) – Gratitude of the patients. (male, 30–34 years old, outpatient sector) Sample description A total of N = 615 observations were used for the analyses. Notably, these observations were obtained from 575 participants (57% male), including eight individuals (1.4%) who participated at all three measurement points and 24 individuals (4.2%) who participated twice. Thus, a total of 543 (94.4%) persons participated only at one of three measurement points, thereby delivering no “truly longitudinal” information on intra-individual change. The participants’ age ranged from <20 to 65–69 years. More than half of the participants (58.4%) were aged between 25 and 44 years. Participants’ work experience ranged from 0 to more than 40 years. In all sectors, roughly half of the participants reported a change of their work area or function due to the pandemic (55% in the hospital sector, 44% in the prehospital sector, and 48% in the outpatient sector). The subsample from the hospital sector (n = 254) included medical doctors, nurses, and undergraduate medical students. The prehospital subsample (n = 250) comprised emergency physicians and paramedics. Regarding the outpatient sector (n = 108), participants were medical doctors, undergraduate medical students, and medical assistants working at general, family or pediatric practices which specialized in the treatment of COVID-19 patients. Three participants did not provide any information about their work sector and were thus excluded from the analysis which focused on differences between work sectors. Private stressors during the pandemic Most of the healthcare professionals did not report private stressors (overall response rate of 29.3%). One third of the given answers was related to the main subject “worries about the health of relatives”. “Caregiving duties” was the second most frequently mentioned theme (S3 Table). In relation to the first main subject (worries about relatives), healthcare professionals mostly mentioned not only relatives and/or friends who belonged to certain risk groups but also stigmatization from family members and friends who were afraid of getting infected. A few participants reported extreme accumulation of family duties or critical life events, such as the dead of a relative or friend. – Wife is afraid of being infected by me. Constant domestic discussions about the dangerous nature of the situation. (male, 65–69 years old, prehospital sector) In relation to the second main subject (caregiving duties), participants most frequently mentioned childcare and home schooling as stressful caregiving duties. – Childcare with 2 full-time working parents with very limited home office facilities. (female, 40–44 years old, prehospital sector) Work stressors during the pandemic Regarding work-related stressors, participants’ ratings on the perceived stressfulness of the 19 pandemic-specific stressors (Table 1) revealed that the necessity of having “strict safety measures” was considered as most stressful. “Fear of getting infected” and “feeling of being isolated from usual work team” were comparably low work stressors. To detect groups of stressors that belong together or co-occurred frequently, we conducted an exploratory factor analyses (EFA) of the 19 distinct stressors. Intercorrelations between the 19 different stressors are presented in Table 1. The EFA revealed four distinct factors (i.e., groups of stressors) because the 4-factorial model demonstrated the best fit with the lowest information criterion BIC (BICs for models with 1 to 9 factors, respectively: 45040.5, 44482.4, 44303.5, 44280.3, 44287.8, 44309.2, 44330.0, 44367.1, 44406.6). In terms of widely used fit indices, this 4-factor model provided good to acceptable fit to the data (RMSEA = 0.06, CFI = 0.947, SRMR = 0.027). Hence, four factors were extracted in total (S1 Table for factor correlations, and S2 Table for factor loadings). In interpreting the core items with highest loadings/factor correlations for each factor (S2 Table), the first factor can be characterized as “fear of transmission” (core items: “fear of infecting others”, λ = 0.898; “fear of infecting friends and family”, λ = 0.881), the second as “interference of workload with private life” (core items: “reduced capacity to use social support due to long work time and stigmatization of healthcare workers”, λ = 0.826; “insufficient capacity to implement basic self-care due to lack of time and energy”, λ = 0.734), and the third as “uncertainty/lack of knowledge” (core items: “information overload of constantly changing information”, λ = 0.740; “no clear instructions”, λ = 0.909). The significance of the fourth factor seems less clear. Whereas even the items with relatively high loadings on this factor show higher loadings on one of the other factors, this fourth factor revealed the lowest correlations with the others. This suggests that this fourth factor reflects some additional source of variance which is largely independent of the other factors, which affects some items that mainly “belong” to one of the first three factors. Notably, the wording of all items loading relatively high on this factor have in common a notion of working within a team (e.g., highest loading core items: “feeling isolated due to separation from usual work team”, λ = 0.339; “stigmatization of oneself and other healthcare personnel working with COVID-19 infected patients”, λ = 0.311). The factor comprises stressors that are likely to affect the respondent more, the more they feel integrated into the work team. The work teams consist of colleagues in the healthcare sector in which the respondents work. Therefore, this factor may be interpreted in terms of “concerns about the team”, a factor which may include both additional stress in terms of responsibility for the team and resilience due to increased team integration. Additional work stressors which were asked about through an open-ended question were mentioned in only 20% of the surveys. “Non-compliance of the society” (12%), “medical supply shortage” (11%) and “increased workload” (11%) were mentioned most frequently. Psychological stress The results showed that “interference of workload with private life” was the pivotal predictor of psychological stress, whereas “concerns about the team” had stress-reducing effects. Generally, stressors had equal effects on psychological stress across work sectors. We now present the results in detail. Results from the structural equation models M1 and M2 (estimates of intercepts and path coefficients) are summarized in Table 2 (see Fig 1 for an illustration of the models) below. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Structural equation model results: estimates of factor means, intercepts, and path coefficients from models 1 and 2. https://doi.org/10.1371/journal.pone.0261502.t002 Overall Model (M1): The factor “fear of transmission” did not reveal any significant effects predicting outcomes of energy, valence, calmness, stress, and fatigue (all ps > .09). The factor "uncertainty/lack of knowledge" showed only small effects predicting these outcomes, with absolute coefficient values ranging from 0.07 to 0.15 (although it was significant at p < .05 in predicting valence and calmness). The factor “interference of workload with private life” was found to be the pivotal factor in predicting low energy (b = -0.51, p < .001), valence (b = -0.38, p < .001), calmness (b = -0.44, p < .001), and high levels of stress (b = 0.53, p < .001), and fatigue (b = 0.51, p < .001). In contrast, the factor “concerns about the team” showed some smaller effects, predicting higher energy (b = 0.219, p < .001) and calmness (b = 0.161, p < .01) and less stress (b = -0.153, p < .01) and fatigue (b = -0.190, p < .001). M1 fitted the data well (RMSEA = 0.035, CFI = 0.967, SRMR = 0.057). Work Sector (M2): The overall test for the equality of all coefficients across the three sectors did not show a significant result (W[40] = 50.99, p = .11), indicating that between-sector differences in the path coefficients could be constrained equally without loss of model fit. Thus, the effects of the identified factors on the outcomes did not differ substantially between the work sectors. Regarding the factor means, the factor “concerns about the team” was significantly higher in the prehospital sector than in the hospital sector. There were no other between-sector mean differences. Notably, however, the outcome intercepts differed between the sectors (overall Wald test: W[10] = 23.22, p = .01). Further tests on each of the outcomes showed that these between-sector differences are significant for calmness (W[2] = 15.01, p < .001), stress (W[2] = 10.30, p = .006) and fatigue (W[2] = 14.37, p < .001), and without α-adjustment for multiple testing also for valence (W[2] = 6.34, p = .04). Thus, as intercept differences could be taken as an indicator of direct sector effects on the outcomes (i.e., effects which are unmediated by the stressor factors), respondents from the outpatient group experienced reduced calmness and felt more stressed, whereas the prehospital group reported lower fatigue. M2 fitted slightly worse than M1, but still fairly well (RMSEA = 0.047, CFI = 0.942, SRMR = 0.065). Crisis management Regarding crisis management, healthcare professionals highlighted social distancing and compulsory masks, training, experience and knowledge exchange, and social support as effective coping strategies during the pandemic. The hospital group mainly mentioned organizational measures such as visiting bans as effective crisis measures, whereas the prehospital sector most frequently named governmental measures such as contact restrictions. We will now proceed to describe the results in detail. Descriptive data of the satisfaction ratings for crisis measures and satisfaction with crisis prevention are presented in Table 3. Satisfaction with crisis measures and prevention was rated medium across all sectors on average, with means ranging from 4.23 to 4.54 for crisis measures and from 3.64 to 3.71 for crisis prevention. Main categories of themes derived from the open-ended questions and frequencies of quotations are reported for each sector (S3 Table). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Satisfaction with short-term crisis measures and crisis prevention. https://doi.org/10.1371/journal.pone.0261502.t003 Effective crisis measures. More than half of the healthcare professionals responded to the open-ended question about subjectively perceived effective crisis measures (57.2%). On the governmental level, healthcare professionals mainly perceived social distancing regulations and compulsory oronasal masks for the general population as effective. – Requirements to keep distance to others create awareness of the gravity of the situation. (male, 50–54, hospital sector) – Patients are used to masks, which they now must also wear in ambulances and hospitals in case of emergency. (male, 30–34 years old, prehospital sector) On the organizational level, healthcare professionals from all sectors perceived visiting bans, bans on elective surgery, and supply of protective equipment for healthcare staff as useful. – Visiting bans of relatives in hospitals; much quieter surroundings and less drama (male, 20–24 years old, prehospital sector) – No plannable surgeries, therefore [there is] more staff on [the] intensive care unit. (female, 45–49 years old, hospital sector) It appears that the perceived effectiveness of these regulations varied across sectors. Organizational measures were mostly mentioned in the hospital sector, whereas governmental measures were more often introduced in the prehospital sector. At the same time, approximately a fifth of the given answers were subsumed under the category “no effective crisis measures have been taken”. In the descriptions of the healthcare professionals, different reasons can be identified for this outcome: a) no measures were taken because no measures were necessary, b) measures should have been taken, but they were not, c) measures were taken but were not sufficient or effective, and d) measures that were taken even aggravated the work. a) We have been relatively well prepared by the annual influenza wave. (female, 30–34 years old, outpatient sector) b) Nothing at all. In my opinion the supply of protective masks and gowns was insufficient. (male, 20–24 years old, hospital sector) c) Since unfortunately people are very inconsistent with the “rule”, which is not even controlled properly, I am very angry. (male, 20–24 years old, sector unknown) d) None! I feel no relief from the crisis measures taken. On the contrary, my field of activity has expanded. (male, 55–59 years old, hospital sector) Effective crisis prevention. Half of the healthcare professionals named effective crisis prevention (overall response rate of 51.4%). Healthcare professionals perceived three main factors as effective in crisis prevention (which together made up more than 67% of the answers, S3 Table): (1) aspects of their professional training, (2) general work experience and exchange, and (3) crisis measures implemented at an early stage. Regarding the training of healthcare professionals, participants mostly mentioned general hygienic training (infection protection courses dealing with highly infectious patients), followed by specific hygienic training related to SARS CoV-2 and further vocational training with a focus on crisis management. In particular, they described repetition and regular updating of such courses as key principles that lead to additional safety when dealing with infectious patients. – Re-training on materials and the correct way to put on and take off the infection protection equipment. (female, 20–24 years old, prehospital sector) – A bunch of good training courses held by the employer of the clinic, e.g., a course related to protective clothing as well as additional information via e-mails I received regularly. (female, 60–64 years old, hospital sector) Besides training, healthcare professionals reported work experience and sharing current work experiences with colleagues as an effective resource during the pandemic. Having experience in dealing with highly infectious patients was perceived as helpful. Occasionally, participants reported that experiences and lessons learnt from other viruses (e.g., avian influenza), other pandemics (e.g., swine flu pandemic) or similar contexts (civil protection activities) prepared them for the COVID-19 pandemic. Sharing current work experience with colleagues through active exchange was perceived as effective and seemed to foster team commitment. – Extensive contact with isolated patients even before the pandemic—routine prevents fear! (male, 35–39 years old, prehospital sector) – Teaching units and good exchange among each other during my currently ongoing professional training. (male, 30–34 years old, hospital sector) – Link with Italian colleagues to exchange expertise. (male, 40–44 years old, hospital sector) – We had regular team meetings; our superior kept us up to date with all the news. (female, 50–54 years old, outpatient sector) One fifth of the reports described early-stage crisis measures as effective in helping crisis prevention. Specifically, the flow of relevant information through internal channels of information including emails or intranet was often mentioned. Nationwide daily information channels such as the Robert Koch Institute (RKI, German Federal Authority for Infectious Diseases) or COVID-19-specific podcasts were also named. Furthermore, the formulation of clear guidelines and recommendations for action and the timely provision of protective material were both perceived as helpful. – Information by my superior, information from the clinic management, but also passing on information among the assistants/specialists, information from senior nurses. (female, 40–44 years old, hospital sector) – Daily "employee news" from the management to the employees in the form of e-mail and information paper at the beginning of the shift. (male, 30–34 years old, hospital sector) – Clear specification of hygiene management in the clinic significantly improved interdisciplinary communication in the clinic and in the department, daily updates by the crisis management team of the clinic and the department also by e-mail to my private e-mail address. – Many visual representations of COVID and hygiene procedures by RKI, but also on the internet by professional associations or Free Open Access Medical Education.  (male, 60–64 years old, hospital sector) – Availability of significantly more protective material! Protective measures before and after each patient contact. (male, 20–24 years old, prehospital sector) It appears that the perceived effectiveness of crisis prevention varied across sectors. General work experience and exchange were mostly mentioned in the hospital and outpatient sectors, whereas early-stage crisis measures were more often applied in the prehospital sector. One seventh of the answers were assigned to the category “no effective crisis prevention has been taken” (see S3 Table). Analogous to the acute crisis measures, different reasons for this answer can be derived from the reports: a) preventive measures should have been taken, but they were not, b) no crisis prevention was possible because the situation was unexpected and dynamic, and, c) the measures taken were insufficient. Irrespective of the judgment of how prepared they felt, some healthcare professionals specified that there was d) no specific crisis management or pandemic-specific training. Additionally, some participants perceived no crisis prevention measures on an organizational level but emphasized the use of individual strategies. a) None. […] Bad (as well as fake) videos of how to put on protective clothing. (male, 25–29 years old, hospital sector) b) None at all. Nobody knew how to deal with trainees or how to deal with the training and further education for rescue service personnel. (female, 35–39 years old, prehospital sector) c) None, nobody knew about the Coronavirus before! (male, 35–39 years old, hospital sector) d) None. Common sense! (male, 25–39 years old, hospital sector) Individual coping. In the open-ended question about individual coping strategies (which had an overall response rate of 83.4%), three areas were frequently mentioned (which together made up 80% of the given answers, S3 Table): (1) social support, (2) hobbies/leisure activities, and (3) mental strategies. Within social contacts, the family is perceived as mainly supportive, while friends and colleagues were also mentioned as important. Most of the named hobbies and leisure activities were sport activities. Walks and opportunities to enjoy nature were also frequently reported. Regarding mental strategies, most participants reported the benefits of distractive activities such as watching videos, listening to audio books, social media, or online shopping. In contrast, other respondents mentioned a deliberately reduced media consumption, although these reports were less frequently noted. The use of relaxation/meditation techniques was often reported as a distraction. Furthermore, a few participants answered with thoughts or plans to quit their job, while others mentioned altruistic or intrinsic motives at work. – I don’t enjoy working for the most part for the first time in my life. (male, 35–39 years old, outpatient sector) – I think about changing my job more often. (male, 40–44 years old, prehospital sector) – The desire to help others with it. (female, 25–29 years old, outpatient sector) – Gratitude of the patients. (male, 30–34 years old, outpatient sector) Effective crisis measures. More than half of the healthcare professionals responded to the open-ended question about subjectively perceived effective crisis measures (57.2%). On the governmental level, healthcare professionals mainly perceived social distancing regulations and compulsory oronasal masks for the general population as effective. – Requirements to keep distance to others create awareness of the gravity of the situation. (male, 50–54, hospital sector) – Patients are used to masks, which they now must also wear in ambulances and hospitals in case of emergency. (male, 30–34 years old, prehospital sector) On the organizational level, healthcare professionals from all sectors perceived visiting bans, bans on elective surgery, and supply of protective equipment for healthcare staff as useful. – Visiting bans of relatives in hospitals; much quieter surroundings and less drama (male, 20–24 years old, prehospital sector) – No plannable surgeries, therefore [there is] more staff on [the] intensive care unit. (female, 45–49 years old, hospital sector) It appears that the perceived effectiveness of these regulations varied across sectors. Organizational measures were mostly mentioned in the hospital sector, whereas governmental measures were more often introduced in the prehospital sector. At the same time, approximately a fifth of the given answers were subsumed under the category “no effective crisis measures have been taken”. In the descriptions of the healthcare professionals, different reasons can be identified for this outcome: a) no measures were taken because no measures were necessary, b) measures should have been taken, but they were not, c) measures were taken but were not sufficient or effective, and d) measures that were taken even aggravated the work. a) We have been relatively well prepared by the annual influenza wave. (female, 30–34 years old, outpatient sector) b) Nothing at all. In my opinion the supply of protective masks and gowns was insufficient. (male, 20–24 years old, hospital sector) c) Since unfortunately people are very inconsistent with the “rule”, which is not even controlled properly, I am very angry. (male, 20–24 years old, sector unknown) d) None! I feel no relief from the crisis measures taken. On the contrary, my field of activity has expanded. (male, 55–59 years old, hospital sector) Effective crisis prevention. Half of the healthcare professionals named effective crisis prevention (overall response rate of 51.4%). Healthcare professionals perceived three main factors as effective in crisis prevention (which together made up more than 67% of the answers, S3 Table): (1) aspects of their professional training, (2) general work experience and exchange, and (3) crisis measures implemented at an early stage. Regarding the training of healthcare professionals, participants mostly mentioned general hygienic training (infection protection courses dealing with highly infectious patients), followed by specific hygienic training related to SARS CoV-2 and further vocational training with a focus on crisis management. In particular, they described repetition and regular updating of such courses as key principles that lead to additional safety when dealing with infectious patients. – Re-training on materials and the correct way to put on and take off the infection protection equipment. (female, 20–24 years old, prehospital sector) – A bunch of good training courses held by the employer of the clinic, e.g., a course related to protective clothing as well as additional information via e-mails I received regularly. (female, 60–64 years old, hospital sector) Besides training, healthcare professionals reported work experience and sharing current work experiences with colleagues as an effective resource during the pandemic. Having experience in dealing with highly infectious patients was perceived as helpful. Occasionally, participants reported that experiences and lessons learnt from other viruses (e.g., avian influenza), other pandemics (e.g., swine flu pandemic) or similar contexts (civil protection activities) prepared them for the COVID-19 pandemic. Sharing current work experience with colleagues through active exchange was perceived as effective and seemed to foster team commitment. – Extensive contact with isolated patients even before the pandemic—routine prevents fear! (male, 35–39 years old, prehospital sector) – Teaching units and good exchange among each other during my currently ongoing professional training. (male, 30–34 years old, hospital sector) – Link with Italian colleagues to exchange expertise. (male, 40–44 years old, hospital sector) – We had regular team meetings; our superior kept us up to date with all the news. (female, 50–54 years old, outpatient sector) One fifth of the reports described early-stage crisis measures as effective in helping crisis prevention. Specifically, the flow of relevant information through internal channels of information including emails or intranet was often mentioned. Nationwide daily information channels such as the Robert Koch Institute (RKI, German Federal Authority for Infectious Diseases) or COVID-19-specific podcasts were also named. Furthermore, the formulation of clear guidelines and recommendations for action and the timely provision of protective material were both perceived as helpful. – Information by my superior, information from the clinic management, but also passing on information among the assistants/specialists, information from senior nurses. (female, 40–44 years old, hospital sector) – Daily "employee news" from the management to the employees in the form of e-mail and information paper at the beginning of the shift. (male, 30–34 years old, hospital sector) – Clear specification of hygiene management in the clinic significantly improved interdisciplinary communication in the clinic and in the department, daily updates by the crisis management team of the clinic and the department also by e-mail to my private e-mail address. – Many visual representations of COVID and hygiene procedures by RKI, but also on the internet by professional associations or Free Open Access Medical Education.  (male, 60–64 years old, hospital sector) – Availability of significantly more protective material! Protective measures before and after each patient contact. (male, 20–24 years old, prehospital sector) It appears that the perceived effectiveness of crisis prevention varied across sectors. General work experience and exchange were mostly mentioned in the hospital and outpatient sectors, whereas early-stage crisis measures were more often applied in the prehospital sector. One seventh of the answers were assigned to the category “no effective crisis prevention has been taken” (see S3 Table). Analogous to the acute crisis measures, different reasons for this answer can be derived from the reports: a) preventive measures should have been taken, but they were not, b) no crisis prevention was possible because the situation was unexpected and dynamic, and, c) the measures taken were insufficient. Irrespective of the judgment of how prepared they felt, some healthcare professionals specified that there was d) no specific crisis management or pandemic-specific training. Additionally, some participants perceived no crisis prevention measures on an organizational level but emphasized the use of individual strategies. a) None. […] Bad (as well as fake) videos of how to put on protective clothing. (male, 25–29 years old, hospital sector) b) None at all. Nobody knew how to deal with trainees or how to deal with the training and further education for rescue service personnel. (female, 35–39 years old, prehospital sector) c) None, nobody knew about the Coronavirus before! (male, 35–39 years old, hospital sector) d) None. Common sense! (male, 25–39 years old, hospital sector) Individual coping. In the open-ended question about individual coping strategies (which had an overall response rate of 83.4%), three areas were frequently mentioned (which together made up 80% of the given answers, S3 Table): (1) social support, (2) hobbies/leisure activities, and (3) mental strategies. Within social contacts, the family is perceived as mainly supportive, while friends and colleagues were also mentioned as important. Most of the named hobbies and leisure activities were sport activities. Walks and opportunities to enjoy nature were also frequently reported. Regarding mental strategies, most participants reported the benefits of distractive activities such as watching videos, listening to audio books, social media, or online shopping. In contrast, other respondents mentioned a deliberately reduced media consumption, although these reports were less frequently noted. The use of relaxation/meditation techniques was often reported as a distraction. Furthermore, a few participants answered with thoughts or plans to quit their job, while others mentioned altruistic or intrinsic motives at work. – I don’t enjoy working for the most part for the first time in my life. (male, 35–39 years old, outpatient sector) – I think about changing my job more often. (male, 40–44 years old, prehospital sector) – The desire to help others with it. (female, 25–29 years old, outpatient sector) – Gratitude of the patients. (male, 30–34 years old, outpatient sector) Discussion Work stressors The present study examined different work stressors and their effects on psychological stress among healthcare professionals during the early stage of the COVID-19 pandemic in Germany. Building upon work stressors which have already been discussed in the literature, we identified four underlying latent stress factors: “fear of transmission”, “interference of workload with private life”, “uncertainty/lack of knowledge” and “concerns about the team”. Among these, “interference of workload with private life” was the pivotal predictor for stress responses. Contrary to many assumptions, the factor "concerns about the team” was associated with a lower psychological stress. Items which were originally intended to assess work stressors seem to have a stress-reducing effect. Potentially the factor “concerns about the team” is an indicator of high team commitment and social support by colleagues, which are well-known stress-buffering factors in the literature [39–41]. Alternatively, individuals who experience less stress may be less “self-focused” and may have more developed capacities to be concerned about others. “Fear of transmission” had no effect and “uncertainty/lack of knowledge” had only minor effects on psychological stress. These unexpected findings call for further investigation of work stressor lists among healthcare professionals during the COVID-19 pandemic. Psychological stress In general, the latent stressors had similar effects on psychological stress across work sectors. Healthcare professionals’ stress and fatigue levels during the COVID-19 pandemic were moderate on average, indicating that they were only remotely impacted during the first wave of the pandemic. Reports of high stress levels are a common finding in studies with frontline healthcare professionals during pandemics such as SARS, Ebola [2, 11–13] and more recently during COVID-19, at least among medical staff working in Asia [2, 4, 23, 42] as well as in Europe [15]. High levels of stress were also anticipated for German medical staff [16]. However, Germany was only moderately affected by the pandemic compared to other European countries [5, 43]. The current finding of moderate stress levels among healthcare professionals seems to mirror this impression. An alternative explanation for this effect lies in the structure of the German healthcare system and the prevention measures taken by the German government (e. g., keeping a large proportion of hospital beds free), which proved to be successful [5], as reflected by the moderate stress levels among the examined hospital staff. There were differences in the stress levels across sectors. The outpatient group was more stressed and less calm, while the prehospital group reported lower fatigue. To better understand these findings, it is necessary to embed them within the specific context of the pandemic situation in Germany. In preparation for the pandemic, considerable burdens on the hospital sector were expected (e.g., scarcity in ICUs; [5]), and so major efforts were targeted at preparing the hospital sector for the pandemic (e.g., clearing wards, generating more ICUs). However, unlike in some neighboring European countries, ICUs in Germany were largely not overcrowded [5] and spare capacities were used to support severely affected neighboring countries. Retrospectively, during the early stage of the COVID-19 pandemic the outpatient sector was affected more severely than the hospital sector, being responsible for screening and testing suspected cases. Health authorities reported and still report that the capacities of outpatient sectors were overstretched in some German regions [44]. Our results mirror this anecdotal impression, since the outpatient group reported that they were more stressed and less calm. This gives reason for concern, since experts expect suspected and infected cases to rise in the flu season, which will again put pressure on outpatient testing resources. While the hospital sector cleared wards to free up ICUs and thereby obviate the anticipated high COVID-19 load, there is currently no possibility of suspending the usual care in the outpatient sector. Given that staff and resources are limited, target-group-specific crisis measures would be helpful to protect healthcare professionals in this sector from severe stress. Thinking beyond upcoming flu seasons, the outpatient sector will also be responsible for additional corona-specific tasks, such as the treatment of an increasing number of patients with "post-COVID-19" complaints and the contribution and administration of future vaccinations. Therefore, in addition to general crisis measures, specifically tailored measures for the outpatient sector should be developed. Furthermore, most COVID-19 patients received treatment in the outpatient sector in Germany. In addition, rapid expansion of hospital capacities including ICU capacities resulted in larger treatment capacities than required. By contrast, outpatient services could frequently not meet the demands. Therefore, when providing psychological support programs to healthcare professionals, one should also be aware of the high stress burden placed on the outpatient sector. Crisis management The analysis of the open-ended questions substantiates the quantitative findings in the study and highlights the differences between the sectors in terms of their perceived effectiveness of the acute crisis management (crisis measures and crisis prevention). Concerning the crisis measures, the hospital sector profited from organizational measures (e.g., visiting bans or bans on elective surgery), whereas the prehospital sector benefitted from governmental measures (e.g., social distancing and compulsory masks for the general population). The outpatient sector reported that both organizational and governmental measures were effective. Concerning crisis prevention, professional training, work experience and exchange, and crisis measures implemented at an early stage were considered as most important. The hospital and outpatient groups benefitted from experience and exchange, while the prehospital sector profited from crisis measures implemented at an early stage. In sum, specific early crisis measures on the governmental level, together with later organizational measures, were successful in reducing healthcare professionals’ stress, and thereby contributed to the protection of healthcare professionals and their mental health, well-being, and functioning. From the results of this study, decision makers in the healthcare sectors should take away two central messages: 1) there should be a priority focus on work stressors related to “interference of workload with private life” in all sectors; and 2) “concerns about the team” potentially driven by high team commitment should be used as a work-specific coping resource that reduces stress responses and improves mental health, well-being, and functioning. In the following, the four latent work stressors, their influence on healthcare professionals’ psychological stress and the distinctive relationships between the sectors are discussed. Finally, specific recommendations for effective crisis management (Fig 2) will be derived for each factor and embedded into the existing research. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Recommendations to strengthen coping resources in health care professionals. Note. a special recommendation for the outpatient sector; b special recommendation for the hospital sector; c special recommendation for the prehospital sector. https://doi.org/10.1371/journal.pone.0261502.g002 “Interference of workload with private life” comprises the higher volume of work, which results in difficulties and reconciliation of work and family as well as insufficient capacities for stress regulation, leading individuals to seek social support or basic self-care. In order to better deal with this work stressor, decision makers should clearly define a feasible workload, while simultaneously supporting private life conduct and strengthening the individual employee’s well-being. Based on the premise that only mentally healthy employees are able to constantly perform at a high level [45, 46], decision makers should implement several steps to limit workload in the context of pandemics. It is essential for healthcare professionals to have enough resting hours at home, appropriate working shifts, and regular breaks [2] in order to enable flexible and family-friendly working hours. There is also a need for preventive concepts for highly affected employees [14]. Furthermore, employers are advised to ensure resting possibilities and opportunities to engage in positive coping strategies during work [26]. With regard to mental health in particular, higher authorities should provide appreciation and positive feedback to employees for their work [14, 43]. Especially in the event that, due to possible staff absences, organizational measures such as sufficient breaks may temporarily not be adhered to, it is important to signal an awareness of this exceptional situation to employees and to appreciate their work so as to strengthen their mood and motivation. Given that healthcare professionals in the outpatient sector seemed to be more affected by the pandemic, specific measures regarding the organization of the outpatient sector in Germany are proposed to reduce their workload. Non-medical tasks should be delegated to other public health institutions, e.g., the supply of protective material, coordination of testing strategy, clear rules of conduct for certificates, and uniform remuneration for COVID-19 services. Processes can be accelerated by reducing bureaucracy, e.g., simplifying the accounting of COVID-19 services. Further preventive measures should include an “opt-out”-regulation of the outpatient workload, in order to avoid compulsory assignment to caring for patients with COVID-19 [2]. Therefore, (triage) guidelines and rules which note the medical interventions that can be skipped or paused in the outpatient sector in the case of increasing COVID-19 cases should be developed as soon as possible. In times of increased workload, social support from family and friends can buffer against psychological stress [2]. Decision makers are responsible for taking various measures to enable social support from family and friends at the workplace, as employees spend an especially large part of their time there during this crisis and are in need of sufficient social support to be able to fulfill their duties and look after their mental health. Social support can be provided by organizational measures, e.g., through the establishment of video facilities for staff during breaks to maintain contact with families and alleviate their concerns [2]. Establishing psychological interventions and education on healthy coping strategies can strengthen well-being, resilience and self-regulation strategies [2, 8]. Low-threshold psycho-social help should also be offered to the healthcare professionals [2, 14]. As an example for best practices, a psycho-social emergency care (“Psychosoziale Notfallversorgung”: PSNV) has been established in Germany. This program assesses concerns and needs, practical support and care, empathic listening, access to information, services and social support, and protection from further harm [14]. Expanding this concept, psycho-social help may also be offered to healthcare professionals’ relatives to support them in supporting the professionals. Nevertheless, the answers to the open-ended questions suggest that healthcare professionals already know stress regulation strategies that work for them. Healthcare professionals reported that the reinforcement of earlier functional coping strategies, such as hobbies and leisure activities or certain adaptive mental strategies (e.g., the use of relaxation or meditation techniques, reduced media consumption or the visualization of reasons to work on the frontline) were effective. Since (mental) health programs and effective individual coping strategies seem to be more effective when ritualized, these measures should be constantly cultivated, especially in non-pandemic times [14]. “Concerns about the team” includes concerns about isolation from colleagues, passing the virus to the workplace and heavier workload for colleagues when falling ill. Interestingly, “concerns about the team” reduced healthcare professionals’ stress responses, indicating that caring for colleagues does not represent a stressor but rather a protective factor. At first glance, this finding may call into question the validity of the work stressors that were empirically tested. However, in interpreting “concerns about the team” as an indicator for high team commitment, decision makers can make use of the protective impact of this factor. In order to better deal with "concerns about the team”, decision makers should pursue the aim of creating a positive corporate culture and promote team building. This quantitative finding is substantiated by the open answers which suggested that sharing (current) work experience with other frontline healthcare professionals was perceived as a pivotal support. Active exchange with colleagues in their own institution, the same region, nationally and internationally made them feel better about working under pandemic circumstances and boosted their confidence at work. Regular team trainings as well as informal team meetings for mutual exchange and a positive safety culture are strategies which build team commitment and offer a conscious reminder of belonging to a team and completing a meaningful task [14]. Through the concept of assertive communication, teams can learn to engage in a more fluid, frank, and direct communication among their members [41]. Given the beneficial influence of “concerns about the team” on stress responses, this finding hints at higher team commitment in the prehospital sector, protecting them from negative health consequences of stress. Therefore, it might be useful to identify team building measures in this sector and to learn from them. At the same time, the qualitative data show that the hospital and the outpatient sector had a higher tendency for exchange between workers from different teams and regions. The lesson learned from this fruitful exchange in the hospital and the outpatient sector is that specific platforms should be created and promoted for all three sectors in order to enable people who have experience in treating similar diseases or situations to train, lead or support the teams who do not have such experiences. “Uncertainty/lack of knowledge” describes the uncertainty in action caused by an information load of constantly changing information, lack of clear instructions and insufficient information about the long-term health consequences of COVID-19. Other facets of this stressor include being insufficiently supplied with personal protective equipment and strict bio-security measures. These facets have also emerged in other work contexts during the COVID-19 pandemic [26]. In order to cope with this work stressor, decision makers should pursue the aim of providing reliable, filtered information and the aim of developing programs to train dealing with uncertainty. For clear communication among healthcare professionals, the subjective reports in our study advocate the involvement of decision makers from all key areas in the crisis management. Knowledge should be shared in regular meetings of representatives for all relevant local public health authorities (e.g., public health department, general practitioners and pediatricians, hospital providers, laboratory managers, and public order department). In this process, a clear distribution of roles and tasks and its constant adaptation, where necessary, might ease communication processes [14]. Based on the information gathered in these meetings, emergency plans and standard operating procedures (SOPs) should be mutually developed [2, 12]. SOPs represent a set of step-by-step instructions, which are designed to help workers carry out complex routine operations. They aim at optimizing efficiency, quality output and performance, while at the same time reducing miscommunication. In future, clear communication structures within these authorities are necessary to make sure that the target-specific information outcomes are transferred to frontline healthcare professionals. In line with previous studies [2, 14], regular professional training and (re-)education about infectious diseases were perceived as particularly effective. “Fear of transmission” mainly captures the fear of infecting others, such as friends, families and colleagues, and the worries about vulnerable family members and friends which is in line with previous work [13, 23]. It also includes the participants’ fear of becoming infected themselves. In order to meet the stressor "fear of transmission”, decision makers should pursue the aim of limiting the risk of infection for healthcare professionals, which in turn will limit the risk of transmission. Limiting the risk of infection for healthcare professionals can be achieved in several ways. One effective way is to provide enough protective clothing so that healthcare professionals are protected when caring for infected patients. Bearing in mind that protective clothing has been scarce at the beginning of the pandemic, which resulted in healthcare professionals having to wear protective clothing designed for one-time-use several times, this way of limiting the risk of infection is crucial. Further, in promoting individual responsibility, healthcare professionals should have easy access to reliable, filtered information and regular hygienic (re-)training. Beyond taking responsibility for oneself, a staff buddy system (which involves double-checking the precise compliance with the hygiene measures by a colleague) is proposed in the literature [2]. Limiting the number of patients who are seen by one healthcare professional per hour may be an additional preventive measure. Potentially, “fear of transmission” is triggered by the stigmatization and exclusion from public life that healthcare professionals had to face during the first wave of the pandemic. Consequently, proper health education for the public and round tables which allow open exchange among healthcare professionals and the community can help to prevent misinformation and stigmatization and, in turn, reduce healthcare professionals’ fear of transmission [47, 48]. Strengths, limitations and future directions Drawing on existing recommendations concerning healthcare professionals’ work stressors during a pandemic [14, 19–21], our study is one of the first empirical investigations of these work stressors during the COVID-19 pandemic in Germany. While there are early studies conducted in Asia on healthcare professionals’ stress responses to the COVID-19 pandemic which concentrate on the outpatient sector [4], the present study included diverse subsamples from all healthcare sectors, including outpatient, prehospital and hospital sectors. The use of open-ended questions enabled us to provide a context to the quantitative analyses, resulting in a more in-depth understanding of the impact of the COVID-19 pandemic on healthcare professionals’ psychological stress experiences. A further advantage of the qualitative supplementary data is that, additionally to a deeper understanding of the individuals themselves, their context and life circumstances can be further investigated. This information is of particular importance for understanding the four main stressors examined in the present study since all four of them relate to the individual’s living environment. Hence, information concerning this environment is particularly useful. At the same time, some limitations apply to the present study. First, the minority of participants (less than 6%) took part in more than one measurement occasion, thereby delivering no truly longitudinal information on intra-individual change. Therefore, the current results should be interpreted as cross-sectional. To increase sample sizes, we distributed the online survey widely across various channels, which reduced the control of the size and demographic characteristics in each subsample. As a consequence, the geographic distribution across Germany and (work) contexts might differ strongly between participants. Medical doctors are also overrepresented in the current sample, which might limit the generalizability to other healthcare professionals. To ensure high participation rates and to keep interference with professional duties to a minimum, we used questionnaires which were originally developed for ecological momentary assessment. Thus, the findings might be limited by the use of single-item measures. Caution must also be taken in the interpretation of the qualitative data: while the response rate for personal helpful coping strategies was high (83.4%), only half of the participants responded to the questions about effective crisis measures and crisis prevention (see also [4]). Having acknowledged the strengths and limitations of the study, its results offer interesting directions for future research. First, the study captured a two-month period at the beginning of the pandemic. Given that epidemiologists already predict further waves, healthcare professionals might continue to be confronted with work stressors related to the COVID-19 pandemic. Our results suggest that, even in the early stage of the pandemic, the perceived severity and impact of the stressors are dynamic. The present study starts at a very early stage of the pandemic when there is virtually no previous experience in dealing with such a situation, enabling it to filter out the true stressors, since no structures have yet been created to facilitate and systematically deal with the situation. The study thus represents a valid starting point for future investigations with regard to stressors. During further outbreaks, it would be interesting to investigate how the four stressors are perceived at a similarly acute time-point, whether new stressors appeared or old ones disappeared and, ultimately, whether following the suggested action guidelines has an impact on the situation at a similar point of time during further waves. Hence, it is necessary to monitor the long-term impact of the pandemic and its related stressors, as chronic stress can have tremendous health consequences [49]. Second, the present study proposes several implications and crisis measures for successfully coping with a pandemic. Future studies should test the efficacy of these measures. Previous studies in healthcare settings [50] and during the pandemic [26] have shown that the use of emotion regulation strategies can help individuals to cope successfully with work stressors and reduce stress responses. While the focus of the present study lays on the identification of stressors, future studies should examine how mechanisms of stress regulation apply to long-lasting, exceptional stress situations such as those experienced during a pandemic. Work stressors The present study examined different work stressors and their effects on psychological stress among healthcare professionals during the early stage of the COVID-19 pandemic in Germany. Building upon work stressors which have already been discussed in the literature, we identified four underlying latent stress factors: “fear of transmission”, “interference of workload with private life”, “uncertainty/lack of knowledge” and “concerns about the team”. Among these, “interference of workload with private life” was the pivotal predictor for stress responses. Contrary to many assumptions, the factor "concerns about the team” was associated with a lower psychological stress. Items which were originally intended to assess work stressors seem to have a stress-reducing effect. Potentially the factor “concerns about the team” is an indicator of high team commitment and social support by colleagues, which are well-known stress-buffering factors in the literature [39–41]. Alternatively, individuals who experience less stress may be less “self-focused” and may have more developed capacities to be concerned about others. “Fear of transmission” had no effect and “uncertainty/lack of knowledge” had only minor effects on psychological stress. These unexpected findings call for further investigation of work stressor lists among healthcare professionals during the COVID-19 pandemic. Psychological stress In general, the latent stressors had similar effects on psychological stress across work sectors. Healthcare professionals’ stress and fatigue levels during the COVID-19 pandemic were moderate on average, indicating that they were only remotely impacted during the first wave of the pandemic. Reports of high stress levels are a common finding in studies with frontline healthcare professionals during pandemics such as SARS, Ebola [2, 11–13] and more recently during COVID-19, at least among medical staff working in Asia [2, 4, 23, 42] as well as in Europe [15]. High levels of stress were also anticipated for German medical staff [16]. However, Germany was only moderately affected by the pandemic compared to other European countries [5, 43]. The current finding of moderate stress levels among healthcare professionals seems to mirror this impression. An alternative explanation for this effect lies in the structure of the German healthcare system and the prevention measures taken by the German government (e. g., keeping a large proportion of hospital beds free), which proved to be successful [5], as reflected by the moderate stress levels among the examined hospital staff. There were differences in the stress levels across sectors. The outpatient group was more stressed and less calm, while the prehospital group reported lower fatigue. To better understand these findings, it is necessary to embed them within the specific context of the pandemic situation in Germany. In preparation for the pandemic, considerable burdens on the hospital sector were expected (e.g., scarcity in ICUs; [5]), and so major efforts were targeted at preparing the hospital sector for the pandemic (e.g., clearing wards, generating more ICUs). However, unlike in some neighboring European countries, ICUs in Germany were largely not overcrowded [5] and spare capacities were used to support severely affected neighboring countries. Retrospectively, during the early stage of the COVID-19 pandemic the outpatient sector was affected more severely than the hospital sector, being responsible for screening and testing suspected cases. Health authorities reported and still report that the capacities of outpatient sectors were overstretched in some German regions [44]. Our results mirror this anecdotal impression, since the outpatient group reported that they were more stressed and less calm. This gives reason for concern, since experts expect suspected and infected cases to rise in the flu season, which will again put pressure on outpatient testing resources. While the hospital sector cleared wards to free up ICUs and thereby obviate the anticipated high COVID-19 load, there is currently no possibility of suspending the usual care in the outpatient sector. Given that staff and resources are limited, target-group-specific crisis measures would be helpful to protect healthcare professionals in this sector from severe stress. Thinking beyond upcoming flu seasons, the outpatient sector will also be responsible for additional corona-specific tasks, such as the treatment of an increasing number of patients with "post-COVID-19" complaints and the contribution and administration of future vaccinations. Therefore, in addition to general crisis measures, specifically tailored measures for the outpatient sector should be developed. Furthermore, most COVID-19 patients received treatment in the outpatient sector in Germany. In addition, rapid expansion of hospital capacities including ICU capacities resulted in larger treatment capacities than required. By contrast, outpatient services could frequently not meet the demands. Therefore, when providing psychological support programs to healthcare professionals, one should also be aware of the high stress burden placed on the outpatient sector. Crisis management The analysis of the open-ended questions substantiates the quantitative findings in the study and highlights the differences between the sectors in terms of their perceived effectiveness of the acute crisis management (crisis measures and crisis prevention). Concerning the crisis measures, the hospital sector profited from organizational measures (e.g., visiting bans or bans on elective surgery), whereas the prehospital sector benefitted from governmental measures (e.g., social distancing and compulsory masks for the general population). The outpatient sector reported that both organizational and governmental measures were effective. Concerning crisis prevention, professional training, work experience and exchange, and crisis measures implemented at an early stage were considered as most important. The hospital and outpatient groups benefitted from experience and exchange, while the prehospital sector profited from crisis measures implemented at an early stage. In sum, specific early crisis measures on the governmental level, together with later organizational measures, were successful in reducing healthcare professionals’ stress, and thereby contributed to the protection of healthcare professionals and their mental health, well-being, and functioning. From the results of this study, decision makers in the healthcare sectors should take away two central messages: 1) there should be a priority focus on work stressors related to “interference of workload with private life” in all sectors; and 2) “concerns about the team” potentially driven by high team commitment should be used as a work-specific coping resource that reduces stress responses and improves mental health, well-being, and functioning. In the following, the four latent work stressors, their influence on healthcare professionals’ psychological stress and the distinctive relationships between the sectors are discussed. Finally, specific recommendations for effective crisis management (Fig 2) will be derived for each factor and embedded into the existing research. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Recommendations to strengthen coping resources in health care professionals. Note. a special recommendation for the outpatient sector; b special recommendation for the hospital sector; c special recommendation for the prehospital sector. https://doi.org/10.1371/journal.pone.0261502.g002 “Interference of workload with private life” comprises the higher volume of work, which results in difficulties and reconciliation of work and family as well as insufficient capacities for stress regulation, leading individuals to seek social support or basic self-care. In order to better deal with this work stressor, decision makers should clearly define a feasible workload, while simultaneously supporting private life conduct and strengthening the individual employee’s well-being. Based on the premise that only mentally healthy employees are able to constantly perform at a high level [45, 46], decision makers should implement several steps to limit workload in the context of pandemics. It is essential for healthcare professionals to have enough resting hours at home, appropriate working shifts, and regular breaks [2] in order to enable flexible and family-friendly working hours. There is also a need for preventive concepts for highly affected employees [14]. Furthermore, employers are advised to ensure resting possibilities and opportunities to engage in positive coping strategies during work [26]. With regard to mental health in particular, higher authorities should provide appreciation and positive feedback to employees for their work [14, 43]. Especially in the event that, due to possible staff absences, organizational measures such as sufficient breaks may temporarily not be adhered to, it is important to signal an awareness of this exceptional situation to employees and to appreciate their work so as to strengthen their mood and motivation. Given that healthcare professionals in the outpatient sector seemed to be more affected by the pandemic, specific measures regarding the organization of the outpatient sector in Germany are proposed to reduce their workload. Non-medical tasks should be delegated to other public health institutions, e.g., the supply of protective material, coordination of testing strategy, clear rules of conduct for certificates, and uniform remuneration for COVID-19 services. Processes can be accelerated by reducing bureaucracy, e.g., simplifying the accounting of COVID-19 services. Further preventive measures should include an “opt-out”-regulation of the outpatient workload, in order to avoid compulsory assignment to caring for patients with COVID-19 [2]. Therefore, (triage) guidelines and rules which note the medical interventions that can be skipped or paused in the outpatient sector in the case of increasing COVID-19 cases should be developed as soon as possible. In times of increased workload, social support from family and friends can buffer against psychological stress [2]. Decision makers are responsible for taking various measures to enable social support from family and friends at the workplace, as employees spend an especially large part of their time there during this crisis and are in need of sufficient social support to be able to fulfill their duties and look after their mental health. Social support can be provided by organizational measures, e.g., through the establishment of video facilities for staff during breaks to maintain contact with families and alleviate their concerns [2]. Establishing psychological interventions and education on healthy coping strategies can strengthen well-being, resilience and self-regulation strategies [2, 8]. Low-threshold psycho-social help should also be offered to the healthcare professionals [2, 14]. As an example for best practices, a psycho-social emergency care (“Psychosoziale Notfallversorgung”: PSNV) has been established in Germany. This program assesses concerns and needs, practical support and care, empathic listening, access to information, services and social support, and protection from further harm [14]. Expanding this concept, psycho-social help may also be offered to healthcare professionals’ relatives to support them in supporting the professionals. Nevertheless, the answers to the open-ended questions suggest that healthcare professionals already know stress regulation strategies that work for them. Healthcare professionals reported that the reinforcement of earlier functional coping strategies, such as hobbies and leisure activities or certain adaptive mental strategies (e.g., the use of relaxation or meditation techniques, reduced media consumption or the visualization of reasons to work on the frontline) were effective. Since (mental) health programs and effective individual coping strategies seem to be more effective when ritualized, these measures should be constantly cultivated, especially in non-pandemic times [14]. “Concerns about the team” includes concerns about isolation from colleagues, passing the virus to the workplace and heavier workload for colleagues when falling ill. Interestingly, “concerns about the team” reduced healthcare professionals’ stress responses, indicating that caring for colleagues does not represent a stressor but rather a protective factor. At first glance, this finding may call into question the validity of the work stressors that were empirically tested. However, in interpreting “concerns about the team” as an indicator for high team commitment, decision makers can make use of the protective impact of this factor. In order to better deal with "concerns about the team”, decision makers should pursue the aim of creating a positive corporate culture and promote team building. This quantitative finding is substantiated by the open answers which suggested that sharing (current) work experience with other frontline healthcare professionals was perceived as a pivotal support. Active exchange with colleagues in their own institution, the same region, nationally and internationally made them feel better about working under pandemic circumstances and boosted their confidence at work. Regular team trainings as well as informal team meetings for mutual exchange and a positive safety culture are strategies which build team commitment and offer a conscious reminder of belonging to a team and completing a meaningful task [14]. Through the concept of assertive communication, teams can learn to engage in a more fluid, frank, and direct communication among their members [41]. Given the beneficial influence of “concerns about the team” on stress responses, this finding hints at higher team commitment in the prehospital sector, protecting them from negative health consequences of stress. Therefore, it might be useful to identify team building measures in this sector and to learn from them. At the same time, the qualitative data show that the hospital and the outpatient sector had a higher tendency for exchange between workers from different teams and regions. The lesson learned from this fruitful exchange in the hospital and the outpatient sector is that specific platforms should be created and promoted for all three sectors in order to enable people who have experience in treating similar diseases or situations to train, lead or support the teams who do not have such experiences. “Uncertainty/lack of knowledge” describes the uncertainty in action caused by an information load of constantly changing information, lack of clear instructions and insufficient information about the long-term health consequences of COVID-19. Other facets of this stressor include being insufficiently supplied with personal protective equipment and strict bio-security measures. These facets have also emerged in other work contexts during the COVID-19 pandemic [26]. In order to cope with this work stressor, decision makers should pursue the aim of providing reliable, filtered information and the aim of developing programs to train dealing with uncertainty. For clear communication among healthcare professionals, the subjective reports in our study advocate the involvement of decision makers from all key areas in the crisis management. Knowledge should be shared in regular meetings of representatives for all relevant local public health authorities (e.g., public health department, general practitioners and pediatricians, hospital providers, laboratory managers, and public order department). In this process, a clear distribution of roles and tasks and its constant adaptation, where necessary, might ease communication processes [14]. Based on the information gathered in these meetings, emergency plans and standard operating procedures (SOPs) should be mutually developed [2, 12]. SOPs represent a set of step-by-step instructions, which are designed to help workers carry out complex routine operations. They aim at optimizing efficiency, quality output and performance, while at the same time reducing miscommunication. In future, clear communication structures within these authorities are necessary to make sure that the target-specific information outcomes are transferred to frontline healthcare professionals. In line with previous studies [2, 14], regular professional training and (re-)education about infectious diseases were perceived as particularly effective. “Fear of transmission” mainly captures the fear of infecting others, such as friends, families and colleagues, and the worries about vulnerable family members and friends which is in line with previous work [13, 23]. It also includes the participants’ fear of becoming infected themselves. In order to meet the stressor "fear of transmission”, decision makers should pursue the aim of limiting the risk of infection for healthcare professionals, which in turn will limit the risk of transmission. Limiting the risk of infection for healthcare professionals can be achieved in several ways. One effective way is to provide enough protective clothing so that healthcare professionals are protected when caring for infected patients. Bearing in mind that protective clothing has been scarce at the beginning of the pandemic, which resulted in healthcare professionals having to wear protective clothing designed for one-time-use several times, this way of limiting the risk of infection is crucial. Further, in promoting individual responsibility, healthcare professionals should have easy access to reliable, filtered information and regular hygienic (re-)training. Beyond taking responsibility for oneself, a staff buddy system (which involves double-checking the precise compliance with the hygiene measures by a colleague) is proposed in the literature [2]. Limiting the number of patients who are seen by one healthcare professional per hour may be an additional preventive measure. Potentially, “fear of transmission” is triggered by the stigmatization and exclusion from public life that healthcare professionals had to face during the first wave of the pandemic. Consequently, proper health education for the public and round tables which allow open exchange among healthcare professionals and the community can help to prevent misinformation and stigmatization and, in turn, reduce healthcare professionals’ fear of transmission [47, 48]. Strengths, limitations and future directions Drawing on existing recommendations concerning healthcare professionals’ work stressors during a pandemic [14, 19–21], our study is one of the first empirical investigations of these work stressors during the COVID-19 pandemic in Germany. While there are early studies conducted in Asia on healthcare professionals’ stress responses to the COVID-19 pandemic which concentrate on the outpatient sector [4], the present study included diverse subsamples from all healthcare sectors, including outpatient, prehospital and hospital sectors. The use of open-ended questions enabled us to provide a context to the quantitative analyses, resulting in a more in-depth understanding of the impact of the COVID-19 pandemic on healthcare professionals’ psychological stress experiences. A further advantage of the qualitative supplementary data is that, additionally to a deeper understanding of the individuals themselves, their context and life circumstances can be further investigated. This information is of particular importance for understanding the four main stressors examined in the present study since all four of them relate to the individual’s living environment. Hence, information concerning this environment is particularly useful. At the same time, some limitations apply to the present study. First, the minority of participants (less than 6%) took part in more than one measurement occasion, thereby delivering no truly longitudinal information on intra-individual change. Therefore, the current results should be interpreted as cross-sectional. To increase sample sizes, we distributed the online survey widely across various channels, which reduced the control of the size and demographic characteristics in each subsample. As a consequence, the geographic distribution across Germany and (work) contexts might differ strongly between participants. Medical doctors are also overrepresented in the current sample, which might limit the generalizability to other healthcare professionals. To ensure high participation rates and to keep interference with professional duties to a minimum, we used questionnaires which were originally developed for ecological momentary assessment. Thus, the findings might be limited by the use of single-item measures. Caution must also be taken in the interpretation of the qualitative data: while the response rate for personal helpful coping strategies was high (83.4%), only half of the participants responded to the questions about effective crisis measures and crisis prevention (see also [4]). Having acknowledged the strengths and limitations of the study, its results offer interesting directions for future research. First, the study captured a two-month period at the beginning of the pandemic. Given that epidemiologists already predict further waves, healthcare professionals might continue to be confronted with work stressors related to the COVID-19 pandemic. Our results suggest that, even in the early stage of the pandemic, the perceived severity and impact of the stressors are dynamic. The present study starts at a very early stage of the pandemic when there is virtually no previous experience in dealing with such a situation, enabling it to filter out the true stressors, since no structures have yet been created to facilitate and systematically deal with the situation. The study thus represents a valid starting point for future investigations with regard to stressors. During further outbreaks, it would be interesting to investigate how the four stressors are perceived at a similarly acute time-point, whether new stressors appeared or old ones disappeared and, ultimately, whether following the suggested action guidelines has an impact on the situation at a similar point of time during further waves. Hence, it is necessary to monitor the long-term impact of the pandemic and its related stressors, as chronic stress can have tremendous health consequences [49]. Second, the present study proposes several implications and crisis measures for successfully coping with a pandemic. Future studies should test the efficacy of these measures. Previous studies in healthcare settings [50] and during the pandemic [26] have shown that the use of emotion regulation strategies can help individuals to cope successfully with work stressors and reduce stress responses. While the focus of the present study lays on the identification of stressors, future studies should examine how mechanisms of stress regulation apply to long-lasting, exceptional stress situations such as those experienced during a pandemic. Conclusion The present study identified four latent work stressors among German healthcare professionals during the COVID-19 pandemic. “Interference of workload with private life” was the pivotal predictor of stress responses, whereas “concerns about the team” had stress-buffering effects. Our findings suggest that the outpatient sector has been affected more severely than the well-prepared hospital sector, since its capacities have been overstretched. In light of a predicted increase to COVID-19 patient load and vaccination duties, specific measures should now be taken to prepare the outpatient sector for the future. To meet the work stressors, healthcare professionals need sector-specific psycho-social support within and outside the workplace that reduces their stress responses and protects their mental health, well-being, and functioning. At the workplace, “concerns about the team” buffers against adverse stress responses, although it involves being more stressed about colleagues’ well-being. As healthcare professionals are predicted to continue to deal with stress as a result of the pandemic in the upcoming months, it is important for decision makers to reduce stress as much as possible, and for healthcare professionals to identify and enforce individual positive coping strategies. Supporting information S1 Table. Factor correlations. https://doi.org/10.1371/journal.pone.0261502.s001 (DOCX) S2 Table. Four-factor solution form the exploratory factor analysis on work-related stressors: Pattern matrix and structure matrix. https://doi.org/10.1371/journal.pone.0261502.s002 (DOCX) S3 Table. Frequencies of personal coping strategies, effective crisis measures and effective crisis prevention across sectors (in % of all coded answers). https://doi.org/10.1371/journal.pone.0261502.s003 (DOCX) Acknowledgments We acknowledge the willing cooperation of all participating healthcare professionals. We would also like to thank the responsible healthcare leaders in each participating institution for their support in distributing the online survey, especially Dr. med. Birgit Abendhard, Prof. Dr. med. Michael Bernhard; PD Dr. med. Christian Bopp and his team; Ralf Dussinger, Hendrik Maier and Tobias Fellhauer; Dr. med. Philipp Gotthardt; Dr. med. Lutz Pollak; Dr. med. Christiane Serf; Dr. med. A. W. and his team; Dr. med. René Wildenauer; Dr. Ines Wolff. We thank Prof. Dr. Ursula Christmann and her team, Prof. Dr. med. Erik Popp, Valerie Schlagenhaufen and Ass Jur. Christoph Wassermann for their valuable support of the study planning. We also acknowledge the great support of Anke Baetzner, Solène Gerwann, Antonia Kind, Carolin Krupop, Anton Linder, Friederike Uhlenbrock and Sonja Steltmann.

Haseeb, Muhammad;Iqbal, Shahid;Hafeez, Muhammad Bilal;Saddiq, Muhammad Sohail;Zahra, Noreen;Raza, Ali;lbrahim, Muhammad Usman;Iqbal, Javaid;Kamran, Muhammad;Ali, Qasim;Javed, Talha;Ali, Hayssam M.;Siddiqui, Manzer H.

doi: 10.1371/journal.pone.0262309pmid: 35025916

1. Introduction Worldwide, dealing with heavy metals toxicity is a major challenge for agricultural scientists because they are in soil environment have become a leading health concern, especially for plants, humans, and animals [1–3]. Due to anthropogenic sources, heavy metals toxicity is increasing in the soil. Among different metals, nickel (Ni) contamination is one of the leading heavy metals that comes from the discharge of effluents from industries, i.e., Ni steel and iron alloys [4], cadmium batteries [5], electroplating [6], and also by the application of pesticide and municipal wastes [7]. Besides these facts, excessive Ni in the land has become a devastating threat to crops’ growth, development, and productivity [8]. Ni toxicity reduced the intake of CO2, declined photosynthesis, chlorophyll contents, and relative water [9], and impaired cell division and elongation [10]. Consequently, the initiation of oxidative damage annoys the balance between antioxidants and reactive oxygen species (ROS) [11], damaging the nucleic acids, proteins, and organelles’ membranes [12]. The management of heavy metals has become crucial to minimize the arising adverse effects on soil, plant, and the environment [2, 13, 14]. Plants have developed metal tolerance mechanisms, chelation, and compartmentalization to maintain trace element homeostasis [15–17]. Therefore, different techniques have been employed in the successful removal of Ni from the soil, such as excavation, electrokinetic, incineration, soil washing (complex, costly, and damage soil quality and fertility), bioremediation, and phytoremediation (ecofriendly and not disturbing the soil fertility and biodiversity) [18–21]. Among them, the growing of hyperaccumulating plant species (phytoremediation) is a good option to cope with elevated Ni in their shoots without expressing toxicity symptoms, and also developed an enzymatic and non-enzymatic antioxidant mechanism which can alleviate oxidative damage to organelles by scavenging ROS species, thereby resulted in higher grain yield [22, 23]. This approach not only removes heavy metals from the soil but also cleans the environment from other pollutants. These heavy metals need to be removed from the soil for agro-ecological sustainability and human benefits. The use of the plant for remediation of heavy metal or phytoextraction is an exciting approach nowadays [24–26]. Phytoextraction requires translocation of heavy metal from contamination surface to harvest stable part [27]. Chenopodium spp. has the genetic ability to accumulate large quantities of heavy metal in leaf tissue. C. quinoa is a better accumulator of Cr, Cd, and Ni and detoxifies contaminated soils [24, 25]. Quinoa has a deep taproot and fibrous root system that allows quinoa plants to access nutrient and soil water unavailable to other plants. These growing characteristics may enhance the uptake efficiency for trace elements [28]. Quinoa is a hyperaccumulator of Pb, Cd, and nickel in the early stages of growth and removes more metals from the soil when grown to maturity [29]. It has gained attention globally due to its high nutritional and health benefits and the ability to grow under contaminated environments [30, 31]. Pakistan is an underdeveloped country, not focusing on the industrial waste containing heavy metals affecting the agricultural soil. Therefore, it is important to determine the morphological and physiological responses, phytoextraction ability, and yield potential of quinoa under Ni-contaminated soils. 2. Materials and methods 2.1. Plant material and Ni imposition A pot trial was executed during winter 2014–15 in a wirehouse (natural environment), Department of Agronomy, University of Agriculture Faisalabad (UAF), Pakistan. Quinoa lines were taken from Alternate Crops Lab, Department of Agronomy, UAF, and the details of lines (A1, A2, A7, and A9) are given in Table 1. Local codes were used to quote respective lines during the study. In order to remove extraneous matter, collected soil samples were air-dried and sieved using a 2 mm sieve and processed to determine physio-chemical characteristics as presented in Table 2. The experiment was carried out in plastic pots (18 cm height and 20 cm base) with 5 kg of the sieved soil. As per the metal concentration of global soils, 0, 50, and 100 mg kg-1 concentrations of Ni were selected [32, 33]. At the same time, 50 and 100 mg kg-1 aqueous solution of Ni was prepared according to respective treatments. The source was analytical grade nickel nitrate Ni(NO3)2 (Sigma-Aldrich), applied to corresponding pots with three replicates. After two months of Ni treatments, seeds of quinoa lines were sown in pots. Fifteen seeds of each quinoa line were sown in each pot. Pots were placed in wire house under ambient light and temperature. The recommended dose of fertilizers (N:P:K @75:60:60 kg ha-1) as urea, diammonium phosphate, and sulfate of potash were applied to each pot. The experiment was conducted using a completely randomized design (CRD) with a factorial arrangement. After emergence, five plants per pot were maintained, and irrigations were applied according to the water requirement of plants. 250 ml distal water per plot was applied daily, and pots were non-leachable. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Detail of quinoa lines used in the current study. https://doi.org/10.1371/journal.pone.0262309.t001 Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Physico-chemical analysis of pot soil used in the current study. https://doi.org/10.1371/journal.pone.0262309.t002 2.2. Growth parameters Eighty days after sowing, plants from each pot were carefully uprooted and separated into shoots and roots by which root is not damaged. Shoot and root length (cm) were measured using a foot ruler. For root and shoot fresh weights (g), plants were washed with a gentle stream of water. After that, their weights were noted with an analytical balance. Shoot and root samples were oven-dried at 70°C for 72 h, then dry shoot and root weights (g) were noted. 2.3. Nickel determination The determination of Ni was done according to the method of Wolf [34], by which 5 mL of HNO3 was taken in each digestion flask containing 0.1 g of dried sample of leaf, root, stem, and seeds (0.1 g). Then incubated the same samples were overnight at room temperature and heated on a hot plate at 250°C until fumes were formed. After that, heating continued again for 30 min and added 1 mL of HNO3 in each flask on cooling and placed back on a hot plate for heating. The same process repeated as described above until the material became clear and colorless. Then, made the volume upto 50 mL using distilled water. The extract was filtered to determine readings using an Atomic Absorption Spectrophotometer (Hitachi Polarized Zeeman AAS, Z8200, Japan) following the procedure as described in [35]. 2.4. Physiological attributes Determinations of Chlorophyll a, b, and carotenoids were measured using the procedure of Arnon [36], and Davis and Goodwin [37], respectively. Briefly, leave samples (0.5 g) were collected at 80 DAS (panical emergence stage) and homogenized in 80% acetone by pestle and mortar under a dark chamber. Then, filtered and made the volume upto 10 mL. Further, 663, 645, and 480 nm UV-V is spectrophotometer (Dynamica Co., UK; Halo DB-20/DB 20S) wavelength was used to determine chlorophyll a, b carotenoids, respectively. The following formula was used for the determination of chlorophyll and carotenoids: Chlorophyll a (mg/g fresh wt.) = (1.27 (OD663)-2.69(OD645) ×V/1000×W Chlorophyll b (mg/g fresh wt.) = (22.9 (OD645)-4.68(OD663) ×V/1000×W Carotenoids (mg/g fresh wt.) = (OD480+0.114(OD663)-0.638 (OD645)/2500) × 1000 For the determination of phenolics, according to Julkunen-Tiitto [38], 80% acetone was used to get an extract of 0.5 g leave sample. Then, the same extract was centrifuged at 12,000 rpm for 5 min, and 100 μL of the extract was mixed in Folin-Ciocalteu’s phenol reagent (0.5 mL), and 2.5 mL of 20% Na2CO3. After that, distilled water was used to make volume upto 5 mL and vortexed for recording absorbance at 750 nm. 2.5. Translocation factor (TF) TF was calculated following the formula of Liu et al. [39]. TF = Metal concentration in shoot (mg/kg)/Metal concentration in root (mg/kg) 2.6. Yield and yield-related attributes Before harvesting, the number of panicles was counted from each pot, and panicle length (cm) was recorded by using a foot ruler. Following the method as described by Jacobsen and Stølen [40], harvested panicles were dried at 25‒30°C using filter paper, and threshing was done manually after ten days. Then, the dry weight and total crop biomass were calculated after sun-drying the samples for a week. Electric balance was used to calculate thousand seed weights. 2.7. Statistical analysis A two-way ANOVA analysis was conducted to analyze the data for a completely randomized design (CRD) that replicated thrice under factorial arrangement with the help of statistical software “Statistics” (ver. 8.1, Tallahassee, FL, USA). R-software (corrplot package) was used to draw correlations among different response variables. 2.1. Plant material and Ni imposition A pot trial was executed during winter 2014–15 in a wirehouse (natural environment), Department of Agronomy, University of Agriculture Faisalabad (UAF), Pakistan. Quinoa lines were taken from Alternate Crops Lab, Department of Agronomy, UAF, and the details of lines (A1, A2, A7, and A9) are given in Table 1. Local codes were used to quote respective lines during the study. In order to remove extraneous matter, collected soil samples were air-dried and sieved using a 2 mm sieve and processed to determine physio-chemical characteristics as presented in Table 2. The experiment was carried out in plastic pots (18 cm height and 20 cm base) with 5 kg of the sieved soil. As per the metal concentration of global soils, 0, 50, and 100 mg kg-1 concentrations of Ni were selected [32, 33]. At the same time, 50 and 100 mg kg-1 aqueous solution of Ni was prepared according to respective treatments. The source was analytical grade nickel nitrate Ni(NO3)2 (Sigma-Aldrich), applied to corresponding pots with three replicates. After two months of Ni treatments, seeds of quinoa lines were sown in pots. Fifteen seeds of each quinoa line were sown in each pot. Pots were placed in wire house under ambient light and temperature. The recommended dose of fertilizers (N:P:K @75:60:60 kg ha-1) as urea, diammonium phosphate, and sulfate of potash were applied to each pot. The experiment was conducted using a completely randomized design (CRD) with a factorial arrangement. After emergence, five plants per pot were maintained, and irrigations were applied according to the water requirement of plants. 250 ml distal water per plot was applied daily, and pots were non-leachable. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Detail of quinoa lines used in the current study. https://doi.org/10.1371/journal.pone.0262309.t001 Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Physico-chemical analysis of pot soil used in the current study. https://doi.org/10.1371/journal.pone.0262309.t002 2.2. Growth parameters Eighty days after sowing, plants from each pot were carefully uprooted and separated into shoots and roots by which root is not damaged. Shoot and root length (cm) were measured using a foot ruler. For root and shoot fresh weights (g), plants were washed with a gentle stream of water. After that, their weights were noted with an analytical balance. Shoot and root samples were oven-dried at 70°C for 72 h, then dry shoot and root weights (g) were noted. 2.3. Nickel determination The determination of Ni was done according to the method of Wolf [34], by which 5 mL of HNO3 was taken in each digestion flask containing 0.1 g of dried sample of leaf, root, stem, and seeds (0.1 g). Then incubated the same samples were overnight at room temperature and heated on a hot plate at 250°C until fumes were formed. After that, heating continued again for 30 min and added 1 mL of HNO3 in each flask on cooling and placed back on a hot plate for heating. The same process repeated as described above until the material became clear and colorless. Then, made the volume upto 50 mL using distilled water. The extract was filtered to determine readings using an Atomic Absorption Spectrophotometer (Hitachi Polarized Zeeman AAS, Z8200, Japan) following the procedure as described in [35]. 2.4. Physiological attributes Determinations of Chlorophyll a, b, and carotenoids were measured using the procedure of Arnon [36], and Davis and Goodwin [37], respectively. Briefly, leave samples (0.5 g) were collected at 80 DAS (panical emergence stage) and homogenized in 80% acetone by pestle and mortar under a dark chamber. Then, filtered and made the volume upto 10 mL. Further, 663, 645, and 480 nm UV-V is spectrophotometer (Dynamica Co., UK; Halo DB-20/DB 20S) wavelength was used to determine chlorophyll a, b carotenoids, respectively. The following formula was used for the determination of chlorophyll and carotenoids: Chlorophyll a (mg/g fresh wt.) = (1.27 (OD663)-2.69(OD645) ×V/1000×W Chlorophyll b (mg/g fresh wt.) = (22.9 (OD645)-4.68(OD663) ×V/1000×W Carotenoids (mg/g fresh wt.) = (OD480+0.114(OD663)-0.638 (OD645)/2500) × 1000 For the determination of phenolics, according to Julkunen-Tiitto [38], 80% acetone was used to get an extract of 0.5 g leave sample. Then, the same extract was centrifuged at 12,000 rpm for 5 min, and 100 μL of the extract was mixed in Folin-Ciocalteu’s phenol reagent (0.5 mL), and 2.5 mL of 20% Na2CO3. After that, distilled water was used to make volume upto 5 mL and vortexed for recording absorbance at 750 nm. 2.5. Translocation factor (TF) TF was calculated following the formula of Liu et al. [39]. TF = Metal concentration in shoot (mg/kg)/Metal concentration in root (mg/kg) 2.6. Yield and yield-related attributes Before harvesting, the number of panicles was counted from each pot, and panicle length (cm) was recorded by using a foot ruler. Following the method as described by Jacobsen and Stølen [40], harvested panicles were dried at 25‒30°C using filter paper, and threshing was done manually after ten days. Then, the dry weight and total crop biomass were calculated after sun-drying the samples for a week. Electric balance was used to calculate thousand seed weights. 2.7. Statistical analysis A two-way ANOVA analysis was conducted to analyze the data for a completely randomized design (CRD) that replicated thrice under factorial arrangement with the help of statistical software “Statistics” (ver. 8.1, Tallahassee, FL, USA). R-software (corrplot package) was used to draw correlations among different response variables. 3. Results 3.1. Effect of Ni on growth parameters All quinoa lines differed significantly for shoot and root length under control and Ni stress conditions. Under maximum Ni application (100 mg kg-1), A1 and A2, unlike other lines, showed a maximum increment in shoot length (2.4 and 1.6%), respectively (Fig 1A). While both these lines showed an increase in root length (6.45 and 2.43%) at a lower Ni dose (50 mg kg-1) as compared to control (Fig 1B). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Influence of different Ni concentrations on growth parameters (a) shoot length; (b) root length; (c) shoot fresh weight; (d) shoot dry weight; (e) root fresh weight; and (f) root dry weight, of four quinoa lines. Error bars denote the standard error of three replications. Bars with the same letters do not differ significantly at p ≤ 0.05. https://doi.org/10.1371/journal.pone.0262309.g001 For shoot dry weight, A7 and A9 exhibited no significant increase between control and 100 mg kg-1 Ni treatment but A1 and A2 lines displayed (2.76 and 11.13%) at 50 mg kg-1 and (0.85 and 7.54%) increment at 100 mg kg-1 Ni application, respectively (Fig 1D). Quinoa A2, unlike other lines, manifested a substantial decline of root dry weight at 50 and 100 mg kg-1 Ni compared to control (Fig 1F). 3.2. Effect of Ni on physiological parameters The photosynthetic pigments of Ni-treated lines were gradually decreased. For instance, compared to control, the maximum reduction of chl a recorded 34, 21.36, 18.86, and 7.86% in A9, A7, A1, and A2 lines, respectively. On the other hand, chl b content decreased 19.53, 16.51, 12.63, and 8.51 in A7, A1, A2, and A9 lines at 100 mg kg-1 Ni application, respectively (Fig 2A and 2B). A minor reduction was observed in carotenoid contents in all lines except A2, in which the values did not significantly reduce (Fig 2C). Compared to other lines, the A2 line showed a higher aptitude to accumulate (57.7%) phenolic at 100 mg kg-1Ni exposure (Fig 2D). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Influence of Ni concentrations on (a) Chl a; (b) Chl b; (c) carotenoids; and (d) soluble phenolic. Error bars denote the standard error of three replications. Bars with the same letters do not differ significantly at p ≤ 0.05. https://doi.org/10.1371/journal.pone.0262309.g002 3.3. Nickel accumulation and translocation This study was determined in terms of the variable phytoextraction potential of four quinoa lines regarding Ni accumulation. The determination made for Ni contents in (leaf, stem, root, and seed) indicated a significant (P<0.01) difference among treatments in quinoa grown in Ni-contaminated pots. Analysis of leaf Ni concentration at all three harvests (multiple leaves stage, panicle emergence stage, and before harvesting) showed a significant increase in Ni concentration in all quinoa lines with an increase in duration and Ni doses. The pattern is followed by quinoa lines (A2>A1>A7>A9) in all three harvests for leaf Ni. At the 2nd harvest (panicle emergence stage), Ni concentration was (10494.7 and 15610.5%) increased in A2 at 50 and 100 mg kg-1, respectively, in comparison to control (Fig 3A). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Nickel concentration in plant parts (a) leaf; (b) stem; and (c) root, and (d) translocation factor. Error bars denote the standard error of three replications. Bars with the same letters do not differ significantly at p ≤ 0.05. https://doi.org/10.1371/journal.pone.0262309.g003 For all quinoa lines, a significant increase in stem Ni concentration was observed when soil Ni was increased from 50 mg kg-1 to 100 mg kg-1. In the A9 line, the Ni concentration in the stem did not exceed (3.26 mg kg-1) with 100 mg kg-1 Ni application. However, A1 succeeded in absorbing (3.03 and 5.1 mg kg-1) nickel with (50 and 100 mg kg-1) Ni dose respectively before harvest (Fig 3B). An increase in Ni doses enhanced the root nickel absorbance of quinoa lines, but maximum increment (3125 and 4950%) with use of (50 and 100 mg kg-1) was observed in A7 compared to control. Contrarily, the lowest root Ni concentration was noted in A2 as compared to other lines. Translocation factor (TF) is used as a tool to access the phytoextraction potential of quinoa lines to remediate the Ni-contaminated soil. An increasing trend was noted in all lines with increasing application of Ni level in an external environment (Fig 3C). The value of TF was recorded maximum in A2 at all three harvests as compared to other lines. Nickel application enhanced translocation value from 0.60 (control) up to 3.09 and 3.21 at (50 and 100 mg kg-1) Ni doses, respectively, at panicle emergence stage (80 days after sowing) in comparison to data of other two harvests (40 and 120 days after sowing) (Fig 3D). The Ni translocation from shoot to seed was observed in all quinoa lines. Maximum seed Ni was found in A7 (1036.5 and 1729.2%) with an application of (50 and 100 mg kg-1) Ni, respectively, compared to control. The trend of quinoa lines regarding Ni storage in seed was A7>A2>A9>A1 (Fig 4E). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. Influence of Ni concentrations on (a) panicle length; (b) the number of panicle; (c) biological yield; (d) seed yield; (e) 1000-seed weight; and (f) seed nickel of four quinoa lines. Error bars denote the standard error of three replications. Bars with the same letters do not differ significantly at p ≤ 0.05. Add the meaning of different lower case letters. https://doi.org/10.1371/journal.pone.0262309.g004 3.4. Effect of Ni on yield-related parameters Applied Ni slightly increased panicle length (3.74 and 3.44%) in A1 and A2 lines respectively at 50 mg kg-1, whereas high application (100 mg kg-1) improved panicle length (3.44%) in the A2 line. The data indicated that Ni application improved (6.45 and 3.22%) panicles number in A2 at 50 and 100 mg kg-1 nickel, respectively. In the A2 line, the biological and seed yield increased at 50 mg kg-1, then a slight reduction was observed at (100 mg kg-1) Ni, while A7 and A9 displayed a decline in these character at both (50 and 100 mg kg-1) Ni levels (Fig 4). 3.5. Correlation analysis A Pearson’s correlation analysis was performed between diverse studied parameters of quinoa under different Ni concentrations (Fig 5). Under 0 mg kg-1 Ni level, LNi was negatively correlated with SL and RL. Further, Niseed, BY, SY SNi were negatively correlated with LNi. At the same time, all other parameters were positively correlated with each other (Fig 5A). In response to 50 mg kg-1 Ni, RNi was negatively correlated with RL, LNi, and SNi. Likewise, Niseed, BY, and SY were negatively correlated with RNi; however, all other parameters were positively correlated (Fig 5B). In response to 100 mg kg-1 Ni, RNi and SNi were negatively correlated with SL, RL, and LNi. Moreover, Niseed, BY and SY were also negatively correlated with SNi and RNi; nonetheless, other parameters were positively correlated (Fig 5C). Overall, the correlation analysis revealed a strong correlation between SY, BY, Niseed, LNi, RL, and SL induced by different Ni concentrations. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. Pearsons correlation analysis between different studies parameters under (a) 0 mg kg-1 Ni, (b) 50 mg kg-1 Ni, and (c) 100 mg kg-1 Ni concentrations. Correlation with color shows the strength of the connection of all experimental parameters. Blue and red colors indicate positive and negative correlation, respectively. Abbreviations: shoot length (SL), root length (RL), leaf Ni contents (LNi), stem Ni contents (SNi), root Ni contents (RNi), Ni in seed (Niseed), biological yield (BY), and seed yield (SY). https://doi.org/10.1371/journal.pone.0262309.g005 3.1. Effect of Ni on growth parameters All quinoa lines differed significantly for shoot and root length under control and Ni stress conditions. Under maximum Ni application (100 mg kg-1), A1 and A2, unlike other lines, showed a maximum increment in shoot length (2.4 and 1.6%), respectively (Fig 1A). While both these lines showed an increase in root length (6.45 and 2.43%) at a lower Ni dose (50 mg kg-1) as compared to control (Fig 1B). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Influence of different Ni concentrations on growth parameters (a) shoot length; (b) root length; (c) shoot fresh weight; (d) shoot dry weight; (e) root fresh weight; and (f) root dry weight, of four quinoa lines. Error bars denote the standard error of three replications. Bars with the same letters do not differ significantly at p ≤ 0.05. https://doi.org/10.1371/journal.pone.0262309.g001 For shoot dry weight, A7 and A9 exhibited no significant increase between control and 100 mg kg-1 Ni treatment but A1 and A2 lines displayed (2.76 and 11.13%) at 50 mg kg-1 and (0.85 and 7.54%) increment at 100 mg kg-1 Ni application, respectively (Fig 1D). Quinoa A2, unlike other lines, manifested a substantial decline of root dry weight at 50 and 100 mg kg-1 Ni compared to control (Fig 1F). 3.2. Effect of Ni on physiological parameters The photosynthetic pigments of Ni-treated lines were gradually decreased. For instance, compared to control, the maximum reduction of chl a recorded 34, 21.36, 18.86, and 7.86% in A9, A7, A1, and A2 lines, respectively. On the other hand, chl b content decreased 19.53, 16.51, 12.63, and 8.51 in A7, A1, A2, and A9 lines at 100 mg kg-1 Ni application, respectively (Fig 2A and 2B). A minor reduction was observed in carotenoid contents in all lines except A2, in which the values did not significantly reduce (Fig 2C). Compared to other lines, the A2 line showed a higher aptitude to accumulate (57.7%) phenolic at 100 mg kg-1Ni exposure (Fig 2D). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Influence of Ni concentrations on (a) Chl a; (b) Chl b; (c) carotenoids; and (d) soluble phenolic. Error bars denote the standard error of three replications. Bars with the same letters do not differ significantly at p ≤ 0.05. https://doi.org/10.1371/journal.pone.0262309.g002 3.3. Nickel accumulation and translocation This study was determined in terms of the variable phytoextraction potential of four quinoa lines regarding Ni accumulation. The determination made for Ni contents in (leaf, stem, root, and seed) indicated a significant (P<0.01) difference among treatments in quinoa grown in Ni-contaminated pots. Analysis of leaf Ni concentration at all three harvests (multiple leaves stage, panicle emergence stage, and before harvesting) showed a significant increase in Ni concentration in all quinoa lines with an increase in duration and Ni doses. The pattern is followed by quinoa lines (A2>A1>A7>A9) in all three harvests for leaf Ni. At the 2nd harvest (panicle emergence stage), Ni concentration was (10494.7 and 15610.5%) increased in A2 at 50 and 100 mg kg-1, respectively, in comparison to control (Fig 3A). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Nickel concentration in plant parts (a) leaf; (b) stem; and (c) root, and (d) translocation factor. Error bars denote the standard error of three replications. Bars with the same letters do not differ significantly at p ≤ 0.05. https://doi.org/10.1371/journal.pone.0262309.g003 For all quinoa lines, a significant increase in stem Ni concentration was observed when soil Ni was increased from 50 mg kg-1 to 100 mg kg-1. In the A9 line, the Ni concentration in the stem did not exceed (3.26 mg kg-1) with 100 mg kg-1 Ni application. However, A1 succeeded in absorbing (3.03 and 5.1 mg kg-1) nickel with (50 and 100 mg kg-1) Ni dose respectively before harvest (Fig 3B). An increase in Ni doses enhanced the root nickel absorbance of quinoa lines, but maximum increment (3125 and 4950%) with use of (50 and 100 mg kg-1) was observed in A7 compared to control. Contrarily, the lowest root Ni concentration was noted in A2 as compared to other lines. Translocation factor (TF) is used as a tool to access the phytoextraction potential of quinoa lines to remediate the Ni-contaminated soil. An increasing trend was noted in all lines with increasing application of Ni level in an external environment (Fig 3C). The value of TF was recorded maximum in A2 at all three harvests as compared to other lines. Nickel application enhanced translocation value from 0.60 (control) up to 3.09 and 3.21 at (50 and 100 mg kg-1) Ni doses, respectively, at panicle emergence stage (80 days after sowing) in comparison to data of other two harvests (40 and 120 days after sowing) (Fig 3D). The Ni translocation from shoot to seed was observed in all quinoa lines. Maximum seed Ni was found in A7 (1036.5 and 1729.2%) with an application of (50 and 100 mg kg-1) Ni, respectively, compared to control. The trend of quinoa lines regarding Ni storage in seed was A7>A2>A9>A1 (Fig 4E). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. Influence of Ni concentrations on (a) panicle length; (b) the number of panicle; (c) biological yield; (d) seed yield; (e) 1000-seed weight; and (f) seed nickel of four quinoa lines. Error bars denote the standard error of three replications. Bars with the same letters do not differ significantly at p ≤ 0.05. Add the meaning of different lower case letters. https://doi.org/10.1371/journal.pone.0262309.g004 3.4. Effect of Ni on yield-related parameters Applied Ni slightly increased panicle length (3.74 and 3.44%) in A1 and A2 lines respectively at 50 mg kg-1, whereas high application (100 mg kg-1) improved panicle length (3.44%) in the A2 line. The data indicated that Ni application improved (6.45 and 3.22%) panicles number in A2 at 50 and 100 mg kg-1 nickel, respectively. In the A2 line, the biological and seed yield increased at 50 mg kg-1, then a slight reduction was observed at (100 mg kg-1) Ni, while A7 and A9 displayed a decline in these character at both (50 and 100 mg kg-1) Ni levels (Fig 4). 3.5. Correlation analysis A Pearson’s correlation analysis was performed between diverse studied parameters of quinoa under different Ni concentrations (Fig 5). Under 0 mg kg-1 Ni level, LNi was negatively correlated with SL and RL. Further, Niseed, BY, SY SNi were negatively correlated with LNi. At the same time, all other parameters were positively correlated with each other (Fig 5A). In response to 50 mg kg-1 Ni, RNi was negatively correlated with RL, LNi, and SNi. Likewise, Niseed, BY, and SY were negatively correlated with RNi; however, all other parameters were positively correlated (Fig 5B). In response to 100 mg kg-1 Ni, RNi and SNi were negatively correlated with SL, RL, and LNi. Moreover, Niseed, BY and SY were also negatively correlated with SNi and RNi; nonetheless, other parameters were positively correlated (Fig 5C). Overall, the correlation analysis revealed a strong correlation between SY, BY, Niseed, LNi, RL, and SL induced by different Ni concentrations. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. Pearsons correlation analysis between different studies parameters under (a) 0 mg kg-1 Ni, (b) 50 mg kg-1 Ni, and (c) 100 mg kg-1 Ni concentrations. Correlation with color shows the strength of the connection of all experimental parameters. Blue and red colors indicate positive and negative correlation, respectively. Abbreviations: shoot length (SL), root length (RL), leaf Ni contents (LNi), stem Ni contents (SNi), root Ni contents (RNi), Ni in seed (Niseed), biological yield (BY), and seed yield (SY). https://doi.org/10.1371/journal.pone.0262309.g005 4. Discussion The use of the plant, e.g., quinoa as phytoextraction, is an exciting approach nowadays [24–26]. The present study indicated that four quinoa lines showed differential responses regarding Ni accumulation. Shoot and root length and fresh and dry biomass of A1 and A2 lines were slightly high at 50 mg kg-1 Ni treatment. For most of these parameters, A1 and A2 displayed better growth at 50 and 100 mg kg-1 Ni than other quinoa lines. However, quinoa lines A7 and A9 reduced their growth with increased Ni concentrations and exposure time. It suggests that the A1 and A2 entail some specific mechanisms to endure the Ni toxicity. Quinoa showed growth promotion at a low amount of metals [41]. Since Ni is an essential micronutrient for plants [42], the low amount is very useful and can improve plant growth [43]. It is well known that Ni is an essential metal for plants metabolism as it plays a key role in enzymes synthesis [44]. Chenopodium species have been documented to show differences for accumulating several heavy metals in the aerial parts [24]. For instance, Al-Whaibi, Siddiqui [45] reported that the growth of wheat was decreased with Ni application. In response to Ni contamination, wheat plants showed pitiable root growth and leaf blade reduction that leads to growth inhibition in wheat [46]. Furthermore, Gabbrielli, Pandolfini [47] reported that growth reduction under Ni treatment is generally related to loss of cellular turgor resulting in inhibit mitotic activity and delay of cell elongation. Nickel can compete with other metals in absorbance due to similar characteristics of nickel with other metals. Therefore, Ni at high concentration inhibits the translocation of other essential metal ions leading to deficiency of other metals in plants [48]. Subsequently, this is the first step in plants to show toxicity and ultimately affect physiological processes [43]. As Ni impedes the translocation of Fe and Mg via competition, this could result in the delay of germination, growth, and yield reduction [49, 50]. Heavy metal toxicity interrupts the biochemical and physiological traits. Production of ROS is the first response to heavy toxicity. Overproduction of ROS results in oxidative stress. However, the secondary sequence of heavy metal stress is a disturbance in the electron transport chain, nutrient homeostasis, and antioxidant system [11, 51]. Heavy metal toxicity lowered the nitrogen, protein content, carotenes, chlorophyll content, and hill reaction [15, 51] as reported in this study that chlorophyll contents were reduced gradually with an increase in Ni concentration in four quinoa lines (Fig 2A and 2D). Haseeb and Maqbool [52] documented the reduction of photosynthetic pigments in sunflower under stress conditions at the reproductive stage. With other heavy metals, Ni is destructive for plant photosynthetic mechanisms and disrupts the electron transport system. Previous reports indicated that Ni mainly accumulates in lamella regions of PSII and hampers photosynthesis [49]. Heavy metal toxicity lowered the nitrogen, protein content, carotenes, chlorophyll content, and hill reaction [24]. A high concentration of heavy metal, e.g., Pb, reduced the growth and photosynthetic pigment. Lead concentration adversely affects the chloroplast structure, which results in reduced enzyme activity, reduced CO2 fixation, and photosynthetic efficiency [24, 25]. In addition, carotenoid has a key role as accessory light-harvesting pigments; accumulation of carotenoid content is also advantageous in annulling oxidative damage caused by heavy metals [53]. It is plausible that greater carotenoid content is also beneficial in abolishing oxidative damage caused by heavy metals and thus protects photosynthetic pigments in the photosystems [53]. Nickel toxicity leads substantial increase in hydroxyl radicles, hydrogen peroxide, and nitric oxide. In response to oxidative stress, plants show induction of enzymatic and non-enzymatic antioxidant defense [54]. Among the non-enzymatic defense, phenolic compounds are important [55, 56]. In this study, data were recorded on the accumulation of soluble phenolic in the control and Ni-treated plants. Reports showed that soluble phenolic is accumulated under Ni toxicity [57]. Metal accumulation leads to an increase in phenolic concentration and maintained membrane integrity [57]. Metal binds with the hydroxyl and carboxyl groups of phenolic and have been identified as metal chelators and thus have a role in minimizing the deleterious effects of metal toxicity [58]. The total amount of Ni absorbed by shoot is considered an important factor in evaluating a plant’s phytoextraction potential. The current study results revealed that All lines accumulated Ni, but A1 and A2 showed more intend to store it in leaf rather than roots without showing any adverse effect in the plant. Heavy metals are absorbed by roots, induce leaf chlorosis, inhibit root growth, and cause the deficiency of essential elements in most plants [15]. Symptom of heavy metal toxicity appeared on young leaves with the formation of dark green ribs. Under the toxicity, leave becomes chlorotic and turns white. Heavy metal toxicity reduced the germination and seedling growth traits, e.g., rice [15] in wheat [59]. Accumulation of a high nickel concentration caused the necrosis and chlorosis in the leaf, i.e., rice. Higher concentration of Ni has impaired membrane, disturbing lipid composition and nutrient homeostasis as Ni toxicity [15]. Nickel was very efficiently sequestered in the leaves of A2 without causing more adverse effects on photosynthetic pigments shows its better tolerance mechanism than other lines. [26] reported that quinoa has a deep taproot system that allows this plant to access soil water and nutrients. This characteristic may enhance the uptake efficiency for trace elements. Quinoa is a hyperaccumulator of Pb, Cd, and nickel in the early stages of growth and removes more metals from the soil when grown to maturity [29]. In the light of previous reports, a possible mechanism adopted by plants for efficient sequestration is the formation of complexes with ligands or compartmentalize into vacuole or cell wall. However, Ni peptide and Ni histidine complexes could also be responsible for Ni transport associated with citrate and malate [60]. Nickel is transported from roots to shoots and leaves through the transpiration stream via the xylem. Yang, Feng [61] explore that cation ATPases or ion channel and cation-proton antiport are involved in xylem loading. Nickel is supplied to meristematic parts of the plants by translocation from old to young leaves, buds, fruits, and seeds, via the phloem. This transport is tightly regulated by metal-ligand complexes and proteins that specifically bind Ni [62]. Yield and yield-related parameters of A1 and A2 lines were slightly increased at low nickel while exposure to high nickel dose gradually reduced yield. Seed yield of A1 and A2 lines were high (17.08 and 15.38%) under 50 mg kg-1 Ni, respectively. Karagiannidis, Stavropoulos [63] reported that tomato yield increased (2.5%) under Ni treatment. It is well known that Ni triggers the catalytic activity of the urease enzyme [44]. A decline in urease activity seriously affects the amino acids and reduces carbon and nitrogen metabolism [64]. For example, Malavolta and Moraes [65] also documented a high yield in soybean under Ni application. It has been reported that exposure of 25 mg kg-1 Ni to the rose of Jamaica elevated fresh and dry biomass. This increment increases the elements like nitrogen, potassium, phosphorus, zinc, and manganese, which had a key role in plant dry matter [66]. 5. Conclusion In conclusion, quinoa A2 is a more efficient phytoremediator of Ni-contaminated soils. Substantial reduction in quinoa growth and yield of quinoa lines A7 and A9 were observed with the increase in Ni concentration. Quinoa A2 line showed high translocation of Ni in shoot without showing any adverse effect on growth. This indicates that A2 has the maximum genetic potential for the safe storage of Ni in the shoot. The possible mechanisms involved are better growth, diverse morpho-anatomical features, Ni sequestration with carotenoids and phenolic, metabolic adjustments, and keeping maximum nutrients in plant parts. Importantly Ni concentrations determined in seed samples of all lines were found below the permissible value set (67 mg kg-1) by FAO/WHO. This information can be used to design novel breeding strategies such as precise backcrossing of these quinoa germplasm by identifying suitable developmental genes into regionally adapted some other cultivars to improve yield under other low-yielding environments to secure the rising global food demand for cereals. Further, the detailed mechanism of Ni toxicity, sequestration, and compartmentalization using other amendments are needed to be investigated by considering the risk assessment of higher toxic levels. Supporting information S1 Data. https://doi.org/10.1371/journal.pone.0262309.s001 (XLSX)

Ahmed, Waqas;Ansari, Shahab;Hanif, Muhammad;Khalil, Akhtar

doi: 10.1371/journal.pone.0262386pmid: 35073373

Introduction Convolutional Neural Networks (CNNs) have achieved state of the art performance in many applications such as face recognition [1], object detection [2], semantic segmentation [3] and other classification tasks. Most of the CNN architectures are deeper and wider and contain a large number of parameters and operations which make them quite difficult to be deployed on power-constrained devices. In most of the cases, these networks are trained and deployed in computationally-rich devices such as graphics processing unit (GPU) for training and multi-core systems for deployment. However, these networks are also being deployed on mobile devices for certain applications such as voice and face recognition etc. In contrast to GPUs and multi-core systems, mobile devices have very limited memory and computational power. The modern deep neural networks are computationally expensive and memory intensive and require more computational power for deployment and training, it has become a challenge to bring the advances in neural network technology to mobile devices. Consequently, much work has been done in recent years, focused on reducing the size of pre-trained neural networks, making them capable to be deployed on mobile devices for inferences [4, 5]. The latest architectures such as inception module [6] or residual connection [7] have millions of parameters which require extensive computation and storage power. Table 1 shows the number of parameters (in millions) for some of the recently proposed CNN architectures. These architectures produce state of the art accuracy and most of the designers start with pre-trained networks for transfer learning purposes. These networks are rarely evaluated on the given datasets and only the classifier is trained and fine-tuned. This usually results in redundancy in the network. Therefore, it is of great importance to devise deep neural network models with relatively low complexity and high accuracy. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Parameters overview of different CNN architectures. https://doi.org/10.1371/journal.pone.0262386.t001 One of the major approaches for the optimization of CNNs is pruning. Pruning is an effective technique to reduce the size of the network by removing redundant filters or weights without effecting the accuracy [8–11]. It has been observed that the filters of well-established CNN models contain redundancy and removing these filters do not cause any degradation in the accuracy of the model. Fig 1 shows the filters of first convolutional layer of AlexNet trained on ImageNet dataset [12]. The redundancy in the filters is clearly visible. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Visualization of first layer convolutional filters of AlexNet. https://doi.org/10.1371/journal.pone.0262386.g001 Pruning can be classified into two categories, i.e. filter pruning [13, 14] and weight pruning [10, 15]. In weight pruning, the weight values which contribute less are directly deleted whereas a whole filter which contributes less is deleted in the filter pruning. Whether it is weight pruning or filter pruning, most of the pruning methods [16–18] involve multiple iterations to identify a suitable threshold for pruning. They start with a pre-trained network and compress the network layers one by one to find out pruning threshold for each layer. This process consumes heavy computational resources and time. Moreover, the network is not fully pruned by any of these approaches alone and redundancy still exists in the network. In order to achieve more pruned network, a mixed approach of filter pruning is presented in this paper where the focus is on redundancy rather than importance of filter towards accuracy. The work of [19] is utilized to compress the network both by reduction in number of layers and reduction in number of filters per layer by using the principal component analysis (PCA). The compressed network is then trained, and some redundant filters are pruned based on the approach given by [20]. The rest of the article is organized as follow. The overview of two step pruning method is given in Section 2. Recent work related to compression of neural networks is discussed in Section 3. In Section 4, the methodology of mixed filter pruning approach is discussed in detail. The results of the proposed scheme are discussed in Section 5. In Section 6, we conclude our work with some future recommendations. CNNs pruning Neural network compression using PCA PCA is used for network analysis to get the compressed design having a fewer number of layers and a fewer number of filters in each layer without any retraining iterations [19]. A pre-trained network is selected and the activations of all layers are analyzed using PCA. The number of filters in each layer of the network is determined by the principal components required to explain 99.9% cumulative variance. The number of layers is determined based on when the number of filters in all layers start contracting. A new network with pre-determined number of convolutional layers and number of filters per layer is constructed and trained to get optimized network. Filter pruning via geometric median As the centrality of data can be best expressed by geometric median, thus it can be utilized to find the filter which minimizes the summation of distances with other filters [20]. Geometric median is used to get common information of all the filters within a single layer. Then all the filters closed to geometric median are identified based on some threshold. As these filters can be represented by other filters in the layer, therefore pruning these has little impact on the network. These filters are then set to zero. The flow chart in Fig 2 shows the overall procedure of two step pruning of a convolutional neural network. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Flowchart of the proposed PCA driven mixed pruning technique. https://doi.org/10.1371/journal.pone.0262386.g002 Neural network compression using PCA PCA is used for network analysis to get the compressed design having a fewer number of layers and a fewer number of filters in each layer without any retraining iterations [19]. A pre-trained network is selected and the activations of all layers are analyzed using PCA. The number of filters in each layer of the network is determined by the principal components required to explain 99.9% cumulative variance. The number of layers is determined based on when the number of filters in all layers start contracting. A new network with pre-determined number of convolutional layers and number of filters per layer is constructed and trained to get optimized network. Filter pruning via geometric median As the centrality of data can be best expressed by geometric median, thus it can be utilized to find the filter which minimizes the summation of distances with other filters [20]. Geometric median is used to get common information of all the filters within a single layer. Then all the filters closed to geometric median are identified based on some threshold. As these filters can be represented by other filters in the layer, therefore pruning these has little impact on the network. These filters are then set to zero. The flow chart in Fig 2 shows the overall procedure of two step pruning of a convolutional neural network. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Flowchart of the proposed PCA driven mixed pruning technique. https://doi.org/10.1371/journal.pone.0262386.g002 Related work In recent years, a number of CNN models are represented having a large number of parameters and memory cost. These models have yield state of the art performance in many applications [21]. However, one of the major drawbacks of these models is over-parameterization which results in heavy computational costs. To tackle this problem, various techniques are proposed to get more robust but computationally efficient models. The work on the acceleration and optimization of CNNs can be divided into four main categories [22], i.e. knowledge distilling, compact convolutional filters, low rank factorization and quantization and pruning. In pruning, multiple techniques were developed to filter out and remove the redundant filters or weights. Some of the main techniques are discussed below. Weight pruning In this type of pruning techniques, the insignificant weights are identified using a predefined criteria and pruned. In [8], a deep compression technique is proposed containing three stages. They first pruned the network while retaining only important weights, then weight sharing was enforced by quantization methods and finally Huffman coding was applied to further reduce the size of the network. They successfully reduced the size of the VGG16 by 49×, however this method requires specialized hardware due to the formation of unstructured sparsity [23]. In [9], the authors suggested an approach for energy-efficient implementation of hardware for large scale neural networks. They have converted a neural network into an approximate neural network by analyzing the importance of a neuron. This was done by using the back propagation mechanism and then subsequently removing the least important connections. This technique is a good approach towards network compression however it requires repeated iterations and consumes huge amount of resources. Similarly, [16, 24] have implemented the weight pruning techniques in a similar manner which results in unstructured sparsity and requires repeated iteration to identify the weights that are less important. Filter pruning In this technique, the entire filter is removed if found insignificant towards the final accuracy of the network. In [25], authors have proposed a scheme to prune the convolutional kernels in neural networks by interleaving criteria-based pruning with fine tuning. They have approximated the changes in the cost function of network parameters by using Taylor expansion. The authors in [11] have proposed the method to prune the redundant filters and retrain the accuracy. They have used the L1-norm to estimate the importance of the filter and filtered out the unimportant filters thus leaving no unstructured sparsity in the pruned network. In [10], authors have removed the entire insignificant filters along with their feature maps to reduce the computational costs significantly. Their work is a good attempt to compress the network and they acheived up to 34× reduction in FLOPs for VGG16 on CIFAR10. Similarly, authors in [26] used the statistics of the next layer to prune the filters in the current layer. These methods do require iterative retraining which consumes heavy computational resources and time. In [25], authors have suggested analysis of all the layers together, however it too requires many repetitive iterations. In [27], authors have removed the filters by introducing multiple losses to each layer. They selected the least important filters by using these losses and the process continues until a stopping criterion meets. This process is done for all layers and it takes multiple iterations. Recently [28] proposed a filter pruning technique based on maximizing the number of zeroes in filters and eliminating the filters having more non-zero parameters. Their idea is based on the fact that if there is any zero value out of two inputs to a multiplier then multiplication operation can be skipped. They are successful in replacing 82% of multiplications with zero-skip multiplications that do not switching energy in multiplication circuits. However, zero-skip multiplication operations are still performed and are not completely eliminated. Weight matrices approximation The third category of techniques to reduce the size and to accelerate the network is the approximation of weight matrices [29–31]. This is done either by quantization or low rank matrix approximation methods. In [32] and [18], authors have used techniques of quantization to reduce the size of the network. Quantization also involves multiple iteration which becomes a bottleneck to large networks. The authors in [33] have proposed to train the CNNs with binary weights and activations. They have divided the network into segments and trained all the segments in a sequential way. The idea of dividing the network into segments is due to the fact that a group can be reconstructed by combining the set of similar binary branches. However, this technique requires custom training methods. In [17], authors have proposed a similar technique to reconstruct the network having fewer parameters however the compression is mainly achieved by regularization. One of the major drawbacks of the regularization process is that it adds more hyper parameters into the process which takes more iterations to achieve optimized model. Learning sparsity patterns The techniques in this category of pruning usually modify the loss function for the model optimization [14]. The weights are detected that minimize the loss while satisfying the pruning criterion. In [34], authors have proposed a method to prune the network during training by using L0-norm regularization in neural networks. The scheme proposed to set the identified weights to zero by including a collection of non-negative stochastic gates. These methods do not guarantee structured sparsity in the network and takes longer training time. The authors in [35] have presented a non-structured architecture that contains a nucleus of connections at the start and the connections are optimized during the training process. This method requires more computational resources to optimize the network. Apart from these four pruning categories, there are some methods that combine different techniques together. In [36], authors have combined weight pruning along with filter pruning to prune the network to the maximum extent. They have used information from the next layer to prune the filters in the current layer. But their method requires a huge amount of computation; first for identifying the insignificant filters and second for the fine tuning to regain accuracy. Weight pruning In this type of pruning techniques, the insignificant weights are identified using a predefined criteria and pruned. In [8], a deep compression technique is proposed containing three stages. They first pruned the network while retaining only important weights, then weight sharing was enforced by quantization methods and finally Huffman coding was applied to further reduce the size of the network. They successfully reduced the size of the VGG16 by 49×, however this method requires specialized hardware due to the formation of unstructured sparsity [23]. In [9], the authors suggested an approach for energy-efficient implementation of hardware for large scale neural networks. They have converted a neural network into an approximate neural network by analyzing the importance of a neuron. This was done by using the back propagation mechanism and then subsequently removing the least important connections. This technique is a good approach towards network compression however it requires repeated iterations and consumes huge amount of resources. Similarly, [16, 24] have implemented the weight pruning techniques in a similar manner which results in unstructured sparsity and requires repeated iteration to identify the weights that are less important. Filter pruning In this technique, the entire filter is removed if found insignificant towards the final accuracy of the network. In [25], authors have proposed a scheme to prune the convolutional kernels in neural networks by interleaving criteria-based pruning with fine tuning. They have approximated the changes in the cost function of network parameters by using Taylor expansion. The authors in [11] have proposed the method to prune the redundant filters and retrain the accuracy. They have used the L1-norm to estimate the importance of the filter and filtered out the unimportant filters thus leaving no unstructured sparsity in the pruned network. In [10], authors have removed the entire insignificant filters along with their feature maps to reduce the computational costs significantly. Their work is a good attempt to compress the network and they acheived up to 34× reduction in FLOPs for VGG16 on CIFAR10. Similarly, authors in [26] used the statistics of the next layer to prune the filters in the current layer. These methods do require iterative retraining which consumes heavy computational resources and time. In [25], authors have suggested analysis of all the layers together, however it too requires many repetitive iterations. In [27], authors have removed the filters by introducing multiple losses to each layer. They selected the least important filters by using these losses and the process continues until a stopping criterion meets. This process is done for all layers and it takes multiple iterations. Recently [28] proposed a filter pruning technique based on maximizing the number of zeroes in filters and eliminating the filters having more non-zero parameters. Their idea is based on the fact that if there is any zero value out of two inputs to a multiplier then multiplication operation can be skipped. They are successful in replacing 82% of multiplications with zero-skip multiplications that do not switching energy in multiplication circuits. However, zero-skip multiplication operations are still performed and are not completely eliminated. Weight matrices approximation The third category of techniques to reduce the size and to accelerate the network is the approximation of weight matrices [29–31]. This is done either by quantization or low rank matrix approximation methods. In [32] and [18], authors have used techniques of quantization to reduce the size of the network. Quantization also involves multiple iteration which becomes a bottleneck to large networks. The authors in [33] have proposed to train the CNNs with binary weights and activations. They have divided the network into segments and trained all the segments in a sequential way. The idea of dividing the network into segments is due to the fact that a group can be reconstructed by combining the set of similar binary branches. However, this technique requires custom training methods. In [17], authors have proposed a similar technique to reconstruct the network having fewer parameters however the compression is mainly achieved by regularization. One of the major drawbacks of the regularization process is that it adds more hyper parameters into the process which takes more iterations to achieve optimized model. Learning sparsity patterns The techniques in this category of pruning usually modify the loss function for the model optimization [14]. The weights are detected that minimize the loss while satisfying the pruning criterion. In [34], authors have proposed a method to prune the network during training by using L0-norm regularization in neural networks. The scheme proposed to set the identified weights to zero by including a collection of non-negative stochastic gates. These methods do not guarantee structured sparsity in the network and takes longer training time. The authors in [35] have presented a non-structured architecture that contains a nucleus of connections at the start and the connections are optimized during the training process. This method requires more computational resources to optimize the network. Apart from these four pruning categories, there are some methods that combine different techniques together. In [36], authors have combined weight pruning along with filter pruning to prune the network to the maximum extent. They have used information from the next layer to prune the filters in the current layer. But their method requires a huge amount of computation; first for identifying the insignificant filters and second for the fine tuning to regain accuracy. Mixed filter pruning Most of the pruning methods involve multiple iterations to identify a suitable threshold for pruning. They start with a pre-trained network and compress the network’s layers one by one to find out pruning threshold for each layer. This process consumes heavy computational resources and time. Moreover, the network is not fully pruned by any of these approaches alone and redundancy still exists in the network. In order to achieve a more optimized network, a 2-step technique of filter pruning is presented in this section. First, PCA is used to analyze the network to get the compressed design having fewer number of layers and fewer number of filters in each layer without any retraining iterations. A pre-trained network is taken, and the activations of all layers are analyzed using PCA. The number of filters in each layer of the network is determined by the principal components required to explain 99.9% cumulative explained variance. The number of layers are determined based on contracting of filters in all layers. A new network with determined number of convolutional layers and number of filters per layer is constructed and trained to get optimized network. Second, geometric median is used to get common information of all the filters within a single layer. Then all the filters are identified which are nearest to geometric median in that layer, based on some threshold. These filters can be represented by other filters in the layer, hence pruning them has little impact on the network. These filters are then removed from the network. Step1: Pruning filters via PCA PCA is used to analyze a pre-trained network without any retraining iterations to get an optimized design. The redundancy in the network is minimized in terms of width (number of filters per layer) and depth (number of layers). The PCA analysis is performed on a pre-trained network by analyzing activations of all the layers simultaneously. The width of the network is determined by the number of principal components required to explain 99.9% cumulative explained variance. Depth of the network is determined based on when the width of the layers starts contracting. In complex data, difficulty in visualization and performing computations on data increases with the increase in the data dimensions. PCA is an analysis technique for the complex data in which only the significant dimensions are retained and the redundant dimensions are removed. For better understanding of the PCA, following terms are defined. Variance is the measurement of variability in the data or how far the data points are with respect to each other. It can be computed as the average squared deviation from the mean as (1) Where xi is the value of one observation x in the ith dimension, is the mean value of all observations and N is the total number of observations. Covariance measures the degree to which corresponding elements from two ordered datasets are moving in the same direction. Positive covariance of two variables x and y indicates that x increases with the increase in y. Negative covariance indicates x in deceasing with the increase in y. Zero covariance indicates that both variables are not related. Covariance can be computed as (2) xi is the value of x in ith dimension Where xi and yi represent the value of variable x and y in the ith dimension; respectively. The and are the mean values of x and y; respectively. N is the total number of values. For a matrix A, an eigenvector is represented by x such that on the multiplication of A and x the direction of the resultant vector is the same as vector x scaled by λ. Mathematically, (3) Here, A is an arbitrary matrix, λ is an eigenvalue of A and x is the eigenvector corresponding to the eigenvalue. A covariance matrix for a dataset having four-dimension a, b, c and d is shown as (4) Where Va represents variance along the dimension a and Ca,b represents covariance along the dimensions a and b. For a matrix X having m × n dimensions where n is the number of data points having m dimension, the covariance matrix can be calculated as (5) Following are the major steps involved in computation of PCA: Calculation of the covariance matrix X for the given data points. Calculation of eigenvectors and corresponding eigenvalues. Sorting of eigenvectors based on their eigenvalues in decreasing order. For a certain threshold k, choose first k eigenvectors which are the new k dimensions. Transformation of the original m dimensional data points into new k dimension. PCA tries to retain high variance along the dimensions, i.e. data to be spread out by removing the correlated dimension. There are two major goals of PCA: (1) to find out linearly independent dimensions so that the data can be represented without any loss, and (2) the original dimension can be reconstructed by those linearly independent dimensions. Once the covariance matrix of the original dataset is calculated, our goal is to transform the original data points so the covariance matrix of the new transformed data points become diagonal matrix. For the m dimensional n data points, the size of covariance matrix is (6) The k principal dimensions are chosen with respect to the k largest eigenvalues. Thus (7) Then input data in reduced dimensions are given as (8) That is, (9) This shows n data points having k dimensions. In other words, (10) Input data to PCA. It is observed that the filters present in the layers of deep convolutional neural networks are highly correlated and contains redundancy. These filters might be detecting the same feature hence making no contribution towards the network performance. In order to reduce the redundancy present in the CNN, activations are used as feature values of the filters. The feature value of a filter is calculated by its convolution with the input patch. The input to PCA is a two-dimensional matrix representing the input sample as its row and corresponding feature value as its column. In the input matrix to PCA, a data point at the location [i, j] is the activation generated by the convolution of ith input patch with jth filter. This input patch convolves with all the filter making a full row of features values. Let AL be the matrix that is obtained by the activations of layer L in a forward pass. The first input patch having size equal to the kernel size convolves with the kernel to produce first pixel of the output activation map. This input patch convolves with M filters to form a row in the output map having dimension R1×1×M. The next input patch is also convolved with all the filters to form another row in the output activation map. The resultant activation map will be having the dimension AL ϵRN × H × W × M, where N is the batch size, H is the height of activation map and W is its width. We can flatten this matrix as (11) where D = N × H × W. The input matrix to the PCA is described in the Fig 3. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Flattened activation matrix as input to PCA. https://doi.org/10.1371/journal.pone.0262386.g003 The flattening of the matrix results into the collection of N × H × W samples in one forward pass. PCA is then performed on BL. Singular Value Decomposition (SVD) is performed on mean normalized symmetric matrix to get M number of eigenvectors Vi and eigenvalues λi. Total variance T is the sum of diagonal elements in the covariance matrix and is equal to the sum of individual variance of the parameters. This is known as trace and is represented as . As trace AB = trace BA, hence trace S = trace W(ΛWT) = trace (ΛWT)W = trace Λ, i.e. trace of BL is sum of its eigenvalues (12) Or (13) Here each eigenvalue is supposed to be explaining λi/T ratio of total variance. As all the eigenvalues are ordered in decreasing order, we can calculate how many cumulative eigenvalues are required to explain 99.9% of the explained total variance. These eigenvalues represent the significant dimensions in the layer and denoted by SL which is given as (14) The significant dimensions are helpful in optimizing the depth and width of the network. Optimizing width and depth of the network. The significant dimensions of the filter space is determined as the number of uncorrelated filters that are able to define 99.9% variance. As discussed above, these PCA matrices are flattened and analyzed by PCA to define the significant dimensions for each layer. These dimensions determine the number of filters per layer in the network. If we consider every layer as a transformation function, in which the expansion of input data into higher dimensions is performed until the data become linearly separable, it implies that the width of the network per layer must be a non-decreasing function. The results show that the number of significant dimension (width of the layer) per layer increase up to certain layer and then starts decreasing. This implies that the layer with lower number of significant dimensions than the preceding layer is contributing no useful transformation towards the final results. Step2: Pruning filters via geometric median Geometric median represents the central tendency in higher dimensions. We can compute the central tendency via geometric median as follow [37]. For a set of n points a1, a2, …, an where every ai ϵRd, find a point xϵRd that minimizes the sum of Euclidian distance to all the points. (15) where (16) For a particular layer L, geometric median can be used to get the common information pertaining to all the filters of L. (17) where xϵRNi × K × K. After finding the geometric median, we can find the filter nearest to geometric median in that layer. (18) where . The filters represented by F(i;j*) are the redundant filters and can be pruned out without any negative effect towards networks output. Now we can determine which filter in the ith layer has the minimum summation of distances with other filters. (19) where xϵ[Fi,1, …, Fi,N+ 1]. Here, the filters represented by F(i, x*) are those filters which share the most common information which means that their information can be replaced by other filters. The effect of removing these filters will be negligible and network can regain the accuracy with fine tuning. Step1: Pruning filters via PCA PCA is used to analyze a pre-trained network without any retraining iterations to get an optimized design. The redundancy in the network is minimized in terms of width (number of filters per layer) and depth (number of layers). The PCA analysis is performed on a pre-trained network by analyzing activations of all the layers simultaneously. The width of the network is determined by the number of principal components required to explain 99.9% cumulative explained variance. Depth of the network is determined based on when the width of the layers starts contracting. In complex data, difficulty in visualization and performing computations on data increases with the increase in the data dimensions. PCA is an analysis technique for the complex data in which only the significant dimensions are retained and the redundant dimensions are removed. For better understanding of the PCA, following terms are defined. Variance is the measurement of variability in the data or how far the data points are with respect to each other. It can be computed as the average squared deviation from the mean as (1) Where xi is the value of one observation x in the ith dimension, is the mean value of all observations and N is the total number of observations. Covariance measures the degree to which corresponding elements from two ordered datasets are moving in the same direction. Positive covariance of two variables x and y indicates that x increases with the increase in y. Negative covariance indicates x in deceasing with the increase in y. Zero covariance indicates that both variables are not related. Covariance can be computed as (2) xi is the value of x in ith dimension Where xi and yi represent the value of variable x and y in the ith dimension; respectively. The and are the mean values of x and y; respectively. N is the total number of values. For a matrix A, an eigenvector is represented by x such that on the multiplication of A and x the direction of the resultant vector is the same as vector x scaled by λ. Mathematically, (3) Here, A is an arbitrary matrix, λ is an eigenvalue of A and x is the eigenvector corresponding to the eigenvalue. A covariance matrix for a dataset having four-dimension a, b, c and d is shown as (4) Where Va represents variance along the dimension a and Ca,b represents covariance along the dimensions a and b. For a matrix X having m × n dimensions where n is the number of data points having m dimension, the covariance matrix can be calculated as (5) Following are the major steps involved in computation of PCA: Calculation of the covariance matrix X for the given data points. Calculation of eigenvectors and corresponding eigenvalues. Sorting of eigenvectors based on their eigenvalues in decreasing order. For a certain threshold k, choose first k eigenvectors which are the new k dimensions. Transformation of the original m dimensional data points into new k dimension. PCA tries to retain high variance along the dimensions, i.e. data to be spread out by removing the correlated dimension. There are two major goals of PCA: (1) to find out linearly independent dimensions so that the data can be represented without any loss, and (2) the original dimension can be reconstructed by those linearly independent dimensions. Once the covariance matrix of the original dataset is calculated, our goal is to transform the original data points so the covariance matrix of the new transformed data points become diagonal matrix. For the m dimensional n data points, the size of covariance matrix is (6) The k principal dimensions are chosen with respect to the k largest eigenvalues. Thus (7) Then input data in reduced dimensions are given as (8) That is, (9) This shows n data points having k dimensions. In other words, (10) Input data to PCA. It is observed that the filters present in the layers of deep convolutional neural networks are highly correlated and contains redundancy. These filters might be detecting the same feature hence making no contribution towards the network performance. In order to reduce the redundancy present in the CNN, activations are used as feature values of the filters. The feature value of a filter is calculated by its convolution with the input patch. The input to PCA is a two-dimensional matrix representing the input sample as its row and corresponding feature value as its column. In the input matrix to PCA, a data point at the location [i, j] is the activation generated by the convolution of ith input patch with jth filter. This input patch convolves with all the filter making a full row of features values. Let AL be the matrix that is obtained by the activations of layer L in a forward pass. The first input patch having size equal to the kernel size convolves with the kernel to produce first pixel of the output activation map. This input patch convolves with M filters to form a row in the output map having dimension R1×1×M. The next input patch is also convolved with all the filters to form another row in the output activation map. The resultant activation map will be having the dimension AL ϵRN × H × W × M, where N is the batch size, H is the height of activation map and W is its width. We can flatten this matrix as (11) where D = N × H × W. The input matrix to the PCA is described in the Fig 3. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Flattened activation matrix as input to PCA. https://doi.org/10.1371/journal.pone.0262386.g003 The flattening of the matrix results into the collection of N × H × W samples in one forward pass. PCA is then performed on BL. Singular Value Decomposition (SVD) is performed on mean normalized symmetric matrix to get M number of eigenvectors Vi and eigenvalues λi. Total variance T is the sum of diagonal elements in the covariance matrix and is equal to the sum of individual variance of the parameters. This is known as trace and is represented as . As trace AB = trace BA, hence trace S = trace W(ΛWT) = trace (ΛWT)W = trace Λ, i.e. trace of BL is sum of its eigenvalues (12) Or (13) Here each eigenvalue is supposed to be explaining λi/T ratio of total variance. As all the eigenvalues are ordered in decreasing order, we can calculate how many cumulative eigenvalues are required to explain 99.9% of the explained total variance. These eigenvalues represent the significant dimensions in the layer and denoted by SL which is given as (14) The significant dimensions are helpful in optimizing the depth and width of the network. Optimizing width and depth of the network. The significant dimensions of the filter space is determined as the number of uncorrelated filters that are able to define 99.9% variance. As discussed above, these PCA matrices are flattened and analyzed by PCA to define the significant dimensions for each layer. These dimensions determine the number of filters per layer in the network. If we consider every layer as a transformation function, in which the expansion of input data into higher dimensions is performed until the data become linearly separable, it implies that the width of the network per layer must be a non-decreasing function. The results show that the number of significant dimension (width of the layer) per layer increase up to certain layer and then starts decreasing. This implies that the layer with lower number of significant dimensions than the preceding layer is contributing no useful transformation towards the final results. Input data to PCA. It is observed that the filters present in the layers of deep convolutional neural networks are highly correlated and contains redundancy. These filters might be detecting the same feature hence making no contribution towards the network performance. In order to reduce the redundancy present in the CNN, activations are used as feature values of the filters. The feature value of a filter is calculated by its convolution with the input patch. The input to PCA is a two-dimensional matrix representing the input sample as its row and corresponding feature value as its column. In the input matrix to PCA, a data point at the location [i, j] is the activation generated by the convolution of ith input patch with jth filter. This input patch convolves with all the filter making a full row of features values. Let AL be the matrix that is obtained by the activations of layer L in a forward pass. The first input patch having size equal to the kernel size convolves with the kernel to produce first pixel of the output activation map. This input patch convolves with M filters to form a row in the output map having dimension R1×1×M. The next input patch is also convolved with all the filters to form another row in the output activation map. The resultant activation map will be having the dimension AL ϵRN × H × W × M, where N is the batch size, H is the height of activation map and W is its width. We can flatten this matrix as (11) where D = N × H × W. The input matrix to the PCA is described in the Fig 3. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Flattened activation matrix as input to PCA. https://doi.org/10.1371/journal.pone.0262386.g003 The flattening of the matrix results into the collection of N × H × W samples in one forward pass. PCA is then performed on BL. Singular Value Decomposition (SVD) is performed on mean normalized symmetric matrix to get M number of eigenvectors Vi and eigenvalues λi. Total variance T is the sum of diagonal elements in the covariance matrix and is equal to the sum of individual variance of the parameters. This is known as trace and is represented as . As trace AB = trace BA, hence trace S = trace W(ΛWT) = trace (ΛWT)W = trace Λ, i.e. trace of BL is sum of its eigenvalues (12) Or (13) Here each eigenvalue is supposed to be explaining λi/T ratio of total variance. As all the eigenvalues are ordered in decreasing order, we can calculate how many cumulative eigenvalues are required to explain 99.9% of the explained total variance. These eigenvalues represent the significant dimensions in the layer and denoted by SL which is given as (14) The significant dimensions are helpful in optimizing the depth and width of the network. Optimizing width and depth of the network. The significant dimensions of the filter space is determined as the number of uncorrelated filters that are able to define 99.9% variance. As discussed above, these PCA matrices are flattened and analyzed by PCA to define the significant dimensions for each layer. These dimensions determine the number of filters per layer in the network. If we consider every layer as a transformation function, in which the expansion of input data into higher dimensions is performed until the data become linearly separable, it implies that the width of the network per layer must be a non-decreasing function. The results show that the number of significant dimension (width of the layer) per layer increase up to certain layer and then starts decreasing. This implies that the layer with lower number of significant dimensions than the preceding layer is contributing no useful transformation towards the final results. Step2: Pruning filters via geometric median Geometric median represents the central tendency in higher dimensions. We can compute the central tendency via geometric median as follow [37]. For a set of n points a1, a2, …, an where every ai ϵRd, find a point xϵRd that minimizes the sum of Euclidian distance to all the points. (15) where (16) For a particular layer L, geometric median can be used to get the common information pertaining to all the filters of L. (17) where xϵRNi × K × K. After finding the geometric median, we can find the filter nearest to geometric median in that layer. (18) where . The filters represented by F(i;j*) are the redundant filters and can be pruned out without any negative effect towards networks output. Now we can determine which filter in the ith layer has the minimum summation of distances with other filters. (19) where xϵ[Fi,1, …, Fi,N+ 1]. Here, the filters represented by F(i, x*) are those filters which share the most common information which means that their information can be replaced by other filters. The effect of removing these filters will be negligible and network can regain the accuracy with fine tuning. Results and discussion The PCA driven mixed filter pruning approach was tested on three publicly available datasets and three neural networks. The CIFAR-10 and CIFAR-100 are online available on https://www.cs.toronto.edu/kriz/cifar.htmland ILSVRC2017 is available on https://image-net.org/challenges/LSVRC/2017/. The CIFAR-10 is tested on VGG-16 and AlexNet while CIFAR-100 and ILSVRC2017 are tested on VGG-19. Mixed pruning is applied on trained networks and redundant filters are pruned out. The compressed models are fine tuned for some epochs to regain the accuracy. On the given datasets and networks, mixed pruning has achieved state of the art results. The models are compressed up to 4.85× in number of parameters and 18.56× in operations without any significant loss in accuracy up to 2.65%. Methods PCA driven mixed pruning approach is implemented using PyTocrh [38] library and was trained on NVIDIA Tesla K80 GPU. The training scheme consists of four different levels: Training the network on given data without any pruning. Training with pruning in terms of depth and width contraction of network based on the approach given by [19]. Training with pruning approach given by [20]. Training with filter pruning based mixed approach. The specifications of the networks and datasets used in the experiments are described in Table 2. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Networks and datasets used in the experiments. https://doi.org/10.1371/journal.pone.0262386.t002 VGG-16 on CIFAR-10 The VGG-16 network with batch normalization was trained on CIFAR-10. The activations of a pre-trained VGG-16 on CIFAR-10 were given to PCA as input to detect the significant dimensions. PCA is used as a dimensionality reduction tool. The significant dimensions represent the filters in a convolutional layer that contributes significantly towards the output. The significant dimension for the VGG-16 network on CIFAR-10 dataset is summarized in Table 3. Based on these significant dimensions, the depth of the network is also compressed along with reduction in number of filters per layer. This phenomenon is shown in the Fig 4. It has been observed that the significant dimensions of the network expand up-to certain limit and then starts contracting. Therefore, it can be concluded that the layer having less or same number of filters than the preceding layer do not contribute towards any significant transformation of input data. In this way, the redundant layers are removed from the network. The final network is compressed to six layers only. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. Compressed architecture of VGG-16 on CIFAR-10. https://doi.org/10.1371/journal.pone.0262386.g004 Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Configuration of VGG-16 on CIFAR-10. https://doi.org/10.1371/journal.pone.0262386.t003 After obtaining the optimized architecture, the network is initialized randomly and trained. The trained network is pruned again based on geometric median. In this approach, the redundant filters are detected and removed. With a little fine tuning of 50 epochs, the network regained its accuracy. The final number of parameters and operations of the network are reduced by 3.33 X and 18.56 X respectively. This is shown on Table 4. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 4. Comparison of VGG-16 on CIFAR-10. https://doi.org/10.1371/journal.pone.0262386.t004 AlexNet on CIFAR-10 AlexNet is a smaller CNN architecture compared to VGG style networks and it is not much over-parametrized. For the analysis of redundancy in the network, the activations obtained by convolution of input patches with the filters of every layer are given to PCA. As a dimensionality reduction tool, PCA is used to identify the significant elements from the input matrix which we call significant dimensions. These significant dimensions are then used to reconstruct the network based on the predetermined length and width. The elements of the network other than significant dimensions are considered redundant and can be discarded. By experiments, it has been observed that the redundancy in the newly constructed network based on the significant dimensions is still present. In order to prune the network further, geometric median based pruning technique is applied which further zeroized the redundant filters. The initial configuration, its significant dimensions and final configuration is shown in Table 5. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 5. Configuration of AlexNet on CIFAR-10. https://doi.org/10.1371/journal.pone.0262386.t005 The layer wise significant dimensions are the key factor to determine the length of the network. It can be concluded that the layer having less or same number of filters than the preceding layer do not contribute towards any significant transformation of input data. In this way, the redundant layers are removed from the network. The final network contains four layers. This is shown in Fig 5. The network is initialized randomly based on the compressed architecture obtained from PCA. This network is trained with the accuracy of 85.64% and the trained model is pruned again based on geometric median. Number of parameters in the final model are reduced by 4.85× and operations are reduced by 2.61×. The results are summarized in Table 6. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. Compressed architecture of AlexNet on CIFAR 10. https://doi.org/10.1371/journal.pone.0262386.g005 Download: PPT PowerPoint slide PNG larger image TIFF original image Table 6. Comparison of AlexNet on CIFAR-10. https://doi.org/10.1371/journal.pone.0262386.t006 VGG-19 on CIFAR-100 VGG-19 is a deep convolutional neural network with 16 convolutional layers. This network is composed of sixteen convolutional layers, five pooling layers and three fully connected layers. VGG-19 is comparatively deeper network and it contains more redundancy. There might be a number of filters detecting the same feature and hence making no significant contribution towards the final out-put of the network. Therefore, this network is much overparametrized and can be compressed significantly. For the compression of this model, two step pruning method is proved to be effective. First, the activations of a pre-trained model are given to PCA for the analysis of significant dimensions. The redundant filters are identified by PCA by analyzing the activation maps of a pre-trained VGG-19 model. The analysis shows that the significant dimensions obtained by PCA are fluctuating across the length of the network. Those layers which have fewer or equal numbers of filters than their preceding layers are treated as insignificant and can be removed from the network as shown in Fig 6. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 6. Compressed architecture of VGG-19 on CIFAR 100. https://doi.org/10.1371/journal.pone.0262386.g006 Once the network is compressed according to the significant dimensions identified by PCA, it is initialized randomly and trained again on the same dataset. Once the required accuracy is obtained, the network is analyzed again for redundancy by using geometric median. To eliminate the redundant filters, filter pruning via geometric median is applied on the trained network and some unimportant filters are set to zero. The final configuration of the network is shown in Table 7. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 7. Configuration of VGG-19 on CIFAR-100. https://doi.org/10.1371/journal.pone.0262386.t007 The results show significant improvement in the network efficiency in terms of number of operations and parameters without any significant loss in accuracy. The FLOPs are reduced by 16.02× and parameters are reduced by 36.4× with a total loss of 2.65% in accuracy. The results are shown in Table 8. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 8. Comparison of VGG-19 on CIFAR-100. https://doi.org/10.1371/journal.pone.0262386.t008 VGG-19 on ILSVRC2017 The final experiment was performed on ILSVRC2017 dataset with batch normalized VGG-19 network. By using the two-step pruning methods, the activations of a pretrained model are given to PCA for the analysis of significant dimensions. The redundant filters are identified by PCA by analyzing the activation maps of a pretrained VGG-19 model. The compressed architecture on the network is analyzed again for redundancy by using geometric median and the redundant filters are zeroized. The summary for the filters in different stages is given in Table 9. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 9. Configuration of VGG-19 on ILSVRC2017. https://doi.org/10.1371/journal.pone.0262386.t009 The compression in terms of number of parameters and number of operations is not as much significant as in case of CIFAR-100 dataset. This is because ILSVRC2017 is a huge dataset and most of the filters tend to learn some features. The Table 10 shows the results of this experiment. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 10. Comparison of VGG-19 on ILSVRC2017. https://doi.org/10.1371/journal.pone.0262386.t010 Methods PCA driven mixed pruning approach is implemented using PyTocrh [38] library and was trained on NVIDIA Tesla K80 GPU. The training scheme consists of four different levels: Training the network on given data without any pruning. Training with pruning in terms of depth and width contraction of network based on the approach given by [19]. Training with pruning approach given by [20]. Training with filter pruning based mixed approach. The specifications of the networks and datasets used in the experiments are described in Table 2. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Networks and datasets used in the experiments. https://doi.org/10.1371/journal.pone.0262386.t002 VGG-16 on CIFAR-10 The VGG-16 network with batch normalization was trained on CIFAR-10. The activations of a pre-trained VGG-16 on CIFAR-10 were given to PCA as input to detect the significant dimensions. PCA is used as a dimensionality reduction tool. The significant dimensions represent the filters in a convolutional layer that contributes significantly towards the output. The significant dimension for the VGG-16 network on CIFAR-10 dataset is summarized in Table 3. Based on these significant dimensions, the depth of the network is also compressed along with reduction in number of filters per layer. This phenomenon is shown in the Fig 4. It has been observed that the significant dimensions of the network expand up-to certain limit and then starts contracting. Therefore, it can be concluded that the layer having less or same number of filters than the preceding layer do not contribute towards any significant transformation of input data. In this way, the redundant layers are removed from the network. The final network is compressed to six layers only. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. Compressed architecture of VGG-16 on CIFAR-10. https://doi.org/10.1371/journal.pone.0262386.g004 Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Configuration of VGG-16 on CIFAR-10. https://doi.org/10.1371/journal.pone.0262386.t003 After obtaining the optimized architecture, the network is initialized randomly and trained. The trained network is pruned again based on geometric median. In this approach, the redundant filters are detected and removed. With a little fine tuning of 50 epochs, the network regained its accuracy. The final number of parameters and operations of the network are reduced by 3.33 X and 18.56 X respectively. This is shown on Table 4. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 4. Comparison of VGG-16 on CIFAR-10. https://doi.org/10.1371/journal.pone.0262386.t004 AlexNet on CIFAR-10 AlexNet is a smaller CNN architecture compared to VGG style networks and it is not much over-parametrized. For the analysis of redundancy in the network, the activations obtained by convolution of input patches with the filters of every layer are given to PCA. As a dimensionality reduction tool, PCA is used to identify the significant elements from the input matrix which we call significant dimensions. These significant dimensions are then used to reconstruct the network based on the predetermined length and width. The elements of the network other than significant dimensions are considered redundant and can be discarded. By experiments, it has been observed that the redundancy in the newly constructed network based on the significant dimensions is still present. In order to prune the network further, geometric median based pruning technique is applied which further zeroized the redundant filters. The initial configuration, its significant dimensions and final configuration is shown in Table 5. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 5. Configuration of AlexNet on CIFAR-10. https://doi.org/10.1371/journal.pone.0262386.t005 The layer wise significant dimensions are the key factor to determine the length of the network. It can be concluded that the layer having less or same number of filters than the preceding layer do not contribute towards any significant transformation of input data. In this way, the redundant layers are removed from the network. The final network contains four layers. This is shown in Fig 5. The network is initialized randomly based on the compressed architecture obtained from PCA. This network is trained with the accuracy of 85.64% and the trained model is pruned again based on geometric median. Number of parameters in the final model are reduced by 4.85× and operations are reduced by 2.61×. The results are summarized in Table 6. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. Compressed architecture of AlexNet on CIFAR 10. https://doi.org/10.1371/journal.pone.0262386.g005 Download: PPT PowerPoint slide PNG larger image TIFF original image Table 6. Comparison of AlexNet on CIFAR-10. https://doi.org/10.1371/journal.pone.0262386.t006 VGG-19 on CIFAR-100 VGG-19 is a deep convolutional neural network with 16 convolutional layers. This network is composed of sixteen convolutional layers, five pooling layers and three fully connected layers. VGG-19 is comparatively deeper network and it contains more redundancy. There might be a number of filters detecting the same feature and hence making no significant contribution towards the final out-put of the network. Therefore, this network is much overparametrized and can be compressed significantly. For the compression of this model, two step pruning method is proved to be effective. First, the activations of a pre-trained model are given to PCA for the analysis of significant dimensions. The redundant filters are identified by PCA by analyzing the activation maps of a pre-trained VGG-19 model. The analysis shows that the significant dimensions obtained by PCA are fluctuating across the length of the network. Those layers which have fewer or equal numbers of filters than their preceding layers are treated as insignificant and can be removed from the network as shown in Fig 6. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 6. Compressed architecture of VGG-19 on CIFAR 100. https://doi.org/10.1371/journal.pone.0262386.g006 Once the network is compressed according to the significant dimensions identified by PCA, it is initialized randomly and trained again on the same dataset. Once the required accuracy is obtained, the network is analyzed again for redundancy by using geometric median. To eliminate the redundant filters, filter pruning via geometric median is applied on the trained network and some unimportant filters are set to zero. The final configuration of the network is shown in Table 7. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 7. Configuration of VGG-19 on CIFAR-100. https://doi.org/10.1371/journal.pone.0262386.t007 The results show significant improvement in the network efficiency in terms of number of operations and parameters without any significant loss in accuracy. The FLOPs are reduced by 16.02× and parameters are reduced by 36.4× with a total loss of 2.65% in accuracy. The results are shown in Table 8. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 8. Comparison of VGG-19 on CIFAR-100. https://doi.org/10.1371/journal.pone.0262386.t008 VGG-19 on ILSVRC2017 The final experiment was performed on ILSVRC2017 dataset with batch normalized VGG-19 network. By using the two-step pruning methods, the activations of a pretrained model are given to PCA for the analysis of significant dimensions. The redundant filters are identified by PCA by analyzing the activation maps of a pretrained VGG-19 model. The compressed architecture on the network is analyzed again for redundancy by using geometric median and the redundant filters are zeroized. The summary for the filters in different stages is given in Table 9. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 9. Configuration of VGG-19 on ILSVRC2017. https://doi.org/10.1371/journal.pone.0262386.t009 The compression in terms of number of parameters and number of operations is not as much significant as in case of CIFAR-100 dataset. This is because ILSVRC2017 is a huge dataset and most of the filters tend to learn some features. The Table 10 shows the results of this experiment. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 10. Comparison of VGG-19 on ILSVRC2017. https://doi.org/10.1371/journal.pone.0262386.t010 Conclusion Deep neural networks have achieved high performance in different data intensive applications. However, their performance comes with a cost of high computational requirement. One possible solution is to compres the network using pruning. Most of the filter pruning techniques are iterative and consumes huge amount of time and computation to compress the network. These techniques have achieved good results however it is observed that the redundancy still exist in the networks and the networks are not fully pruned. A mixed pruning method without the involvement of any repetitive iterations is presented in this paper to remove the redundancy in the convolutional neural networks. First, the trained model is analyzed using PCA and its significant filters are detected by giving the activations as input to PCA. Based on these dimensions, a new network is initialized randomly with a smaller number of layers and smaller number of filters per layer. The newly formed model is trained and analyzed again for redundancy using geometric median. The identified redundant filters are then set to zero. The network is then fine tuned to regain its accuracy with an optimized number of layers. The proposed network pruning method helps significantly in reducing the computational cost of the network but at the same also cause a minimal loss in accuracy. One of the possible future direction may be working on the pruning with improved accuracy.

Vuoskoski, Jonna K.;Zickfeld, Janis H.;Alluri, Vinoo;Moorthigari, Vishnu;Seibt, Beate

doi: 10.1371/journal.pone.0261151pmid: 35020739

Introduction The emotional effects of music are among the most important reasons for engaging in music listening in everyday life (e.g., [1–3]). These effects range from slight changes in affective state to exceptionally strong, transformative experiences [4]. One commonly reported response to music is feeling moved or touched [1]. Beyond the domain of music, this phenomenon has garnered increased interest in recent years. Theoretical arguments and empirical evidence suggest that people often say they are moved or touched in response to increased affiliation and morality, and that the emotional state is experienced as predominantly positive, often features tears, chills, or warm feelings, and motivates social bonding [5, 6] (Zickfeld et al., 2019). This evidence further indicates that people across a vast array of cultures and languages respond similarly to elicitors of the emotional state, labeling their state with corresponding terms in their language. In English, people typically use the terms moved and touched to describe their state. We will use feeling moved to denote this subjective feeling state. However, there exist few systematic studies that have tested the convergence of these features in response to music. Rather, prior studies focused on specific components or certain types of music only (e.g., [7]). Are people moved by the same affiliative aspects when listening to music as they are when for example reuniting with a loved one? What musical properties facilitate feeling moved? There is some evidence that music can indeed convey affiliation motives and that listeners can feel socially connected to different aspects of music or instruments [8–10]. In the present study, we explore correlates of feeling moved or touched in response to music, focusing on the question of whether theories of feeling moved can account for responses to music and which musical and acoustical features contribute to the experience of feeling moved. Instead of probing individuals’ subjective responses after listening to different musical excerpts, we asked participants in the current study to continuously rate their experiences and perceptions while they unfold during music listening using a continuous self-report paradigm (e.g., [11]). Theories of feeling moved Building on anthropological and ethnographic work, Fiske et al. [12] introduced kama muta as a universal tendency to respond emotionally (in a way that is often described as moved or touched) when communal sharing suddenly intensifies. Communal sharing is one of four basic schemes (or models) of relating to others focusing on what we have in common. It is expressed by giving according to need and ability (while the other three models are expressed by giving according to hierarchy, equality and proportionality, respectively), by bodily proximity, touch, synchrony, or food sharing (see [13], for a more detailed introduction of communal sharing and the remaining models). States of kama muta are thus characterized by appraisals of increased interpersonal closeness or connectedness (reflecting the intensification of communal sharing), by labeling them as moved, touched or heartwarming, as well as by sensations of tears, warmth in the chest and chills, by experiencing them as positive, and by renewed devotion to one’s relationships characterized by communal sharing [14, 15]. Empirical evidence has supported kama muta theory across several different cultural contexts [5, 16]. However, so far the empirical studies testing kama muta theory have only used stories and videos to evoke kama muta through the intensification of communal sharing between human characters. Fiske [14] assumes that the same appraisal of increased closeness or connectedness also accounts for instances where people describe being moved by music: “Sometimes the CS that suddenly intensifies is between musicians and audience, sometimes among the musicians or singers, sometimes among the audience, sometimes with the composer or even with the music itself.” However, so far, this proposition has not been tested. Kama muta theory defines the emotion of kama muta through the main appraisal theme, intensification of communal sharing (labelled an elicitor-specific eudaimonic emotion in recent theorizing, [17]). The labels that persons give their emotional experience, such as moved, touched, heartwarming, raptured and others, in English, are but one, though for diagnostic purposes often very important, index for the emotion [12]. Conversely, other theories define the emotion through the label “being moved”, encompassing assumed elicitors, subjective feelings, physiological signs and sensations and often also motivational tendencies (a feeling-specific eudaimonic emotion according to Landmann [17], see [6], for a review). Of these, two are particularly relevant for the present investigation. Konečni [18, 19] treats being moved as part of the aesthetic trinity, next to thrills or chills and states of aesthetic awe. Thereby, being moved represents strong emotional states that are experienced in response to the sublime (including musical stimuli). He contends that chills are, in the context of music, far more frequent and predictable (because they are shallower) than the state of being moved. He considers being moved to be a rarer response and determined by the personal associative context of the person. If this was the case, there should be low interindividual agreement on what the moving segments in a musical piece are. Conversely, kama muta theory suggests that being moved can be evoked reliably by musical passages because the music itself conveys an intensification of communal sharing. We also expect that for moving music, the occurrence of chills and feeling moved or touched coincide to a large degree. The other theory we draw on for the present investigation is the distancing-embracing model [20], which posits that being moved is involved in transforming negative states into pleasurable experiences in aesthetic perceptions, including music listening. Being moved is thereby conceptualized as a mixed emotion that is experienced as predominantly positive. The authors distinguish between two prototypes of being moved: a sadly moving and a joyfully moving variant [21, 22]. They expect that being moved plays a more important role in the enjoyment of sad stimuli. Indeed, empirical research has indicated that being moved is a mediating factor in explaining the enjoyment of artworks that are experienced as sad [22, 23]. Similar findings have also been obtained for sad music, and empathic responses were identified as a possible driver [7, 24]. Empathy, in turn, has been identified as a characteristic of at least an important variant of being moved, and empathic predispositions predict the propensity to feel moved [6, 21, 25]. It remains to be shown, however, whether feeling moved or touched by predominantly sad versus joyful pieces of music have the same profile, which would strengthen the argument that it is the same emotion in different affective contexts. To summarize, based on kama muta theory we predict that feeling moved or touched by music co-occurs with a sense of connectedness and weeping, feelings of warmth in the chest, and chills. While these attributes are compatible with all theories on the state of feeling moved (for overviews, see [6, 26]), the aesthetic trinity theory [18] predicts low agreement regarding which segments move individuals, while kama muta theory predicts high agreement for music that is preselected to be experienced as moving. This high agreement should be caused by similar emotional responses to musical features, and we shall attempt to identify such features. Lastly, we predict that feeling moved by sadly and joyfully moving music will show comparable correlation patterns. This would suggest that it is the same emotion in both cases, whether two variants of the same mixed emotion [21] or the same emotion with different concurrent emotions of sadness and joy [14]. Feeling moved as a response to music As briefly alluded to earlier, feeling moved has received increased theoretical and empirical attention in the context of music. In a questionnaire study a total of 141 participants indicated that feeling moved was their fourth most common emotional response in listening to music [1], and interviews reveal that individuals often say that they are moved in the context of strong and profound experiences with music [4]. Similarly, Scherer and Zentner [27] identified feeling moved as a common response when listening to music, involving symptoms such as moist eyes or chills (although they treated feeling moved as a vague emotional category) and a recent survey showed that feeling moved is the most commonly reported emotional state when investigating crying in response to music [28]. Items assessing feeling moved have been included in the Geneva Emotional Music Scale [29] and a hierarchical clustering approach suggests that feeling moved (as measured by the items moved and touched) form a distinct cluster of musically induced emotions [30], being most similar to the states of wonder and transcendence. In a recent review of factors affecting the enjoyment of sad music, Eerola et al. [24] identified feeling moved as a possible mediator, next to empathy or social surrogacy. Across two experiments Vuoskoski and Eerola [7] examined this proposition empirically. Testing 327 participants across two experiments, the authors presented several musical excerpts that differed in terms of movingness and sadness, and collected responses on felt sadness, feeling moved and liking. In the first experiment, feeling moved fully mediated the relationship between felt sadness and liking (see [23] for similar findings using film stimuli), and the second experiment further confirmed that this relationship could not be explained by the perceived beauty of the musical pieces. Rather, movingness appeared to contribute to the perceived beauty of sad music. Given these theoretical and empirical findings, Zickfeld [31] suggested that kama muta theory might incorporate and explain several factors contributing to the enjoyment of sad music (identified in Eerola et al.’s [24] review): feeling moved, empathy, social surrogacy, or more precisely communal sharing and affiliation, and experiencing sad music as positive and pleasurable. While it is easy to argue that music may increase affiliation through lyrics expressing tenderness and love [12], the ways in which instrumental music is to convey affiliation seem less straightforward. On one hand, a listener might experience an increase in affiliation by identifying or feeling a connection to the music in general (for example due to rhythmic entrainment; see e.g., [32]), the composer, the musicians, or other fans/listeners. Listening to familiar music may also remind us of nostalgic relationships with significant others: Konečni [18] argues that feeling moved by music is individually determined by the associative context or web that one has constructed with the musical piece, possibly also involving an affiliation towards certain aspects of the music. Evidence from an empirical study on nostalgia responses to music supports this assertion [33]. The authors found that musical pieces were rated as more nostalgic if they were perceived as familiar or more autobiographically salient. In addition, there is evidence that listeners can infer affiliative motives from musical improvisations [8], can feel connected to music through a form of empathy [34], and that listening to moving music from a specific culture can increase affiliation towards that culture [35]. On the other hand, in more interactive contexts feeling moved might occur due to concert attendants dancing or moving in synchrony, singing in unison, as well as perceiving the musicians performing music in synchrony. All of these aspects have been shown to elicit social bonding (e.g., [36]), and are part of communal sharing relationships that might intensify in specific contexts [13]. Musical chills. Further, strong emotional responses to music have been associated with the occurrence of chills, goosebumps or frissons [37–42]. Chills are considered a relatively common, positive psychophysiological response to music or aesthetic stimuli (see [37], for a review). Research has sometimes distinguished between chills as a subjective emotional response and goosebumps or piloerection as an objective physiological response [43]. In the present manuscript, we follow Maruskin et al. [44], who argued that goosebumps represent one physiological component of pleasurable chills. Empirical studies have linked the occurrence of chills to strong experiences of feeling moved using qualitative methods, self-report ratings and more observational techniques such as camera devices [5, 22, 38, 43, 45]. Bannister [39] found in a comprehensive survey on chills responses to music that a high number of participants highlighted social aspects such as feelings of connectedness, the human voice, and perceived relationships between virtual agents (i.e., musical instruments/parts) as evoking chills. Similarly, a recent survey study identified moving chills as one of three types of chill responses to aesthetic stimuli, which co-occurred with feeling moved, tenderness, and tears (the other types being warm chills and cold chills, Bannister [38]; see also [44]). Such results fit previous findings and theories on feeling moved and highlight the importance of chills as a correlate of feeling moved or touched in response to music. Feeling moved and musical features To the best of our knowledge, there has been no direct research linking responses of feeling moved by music to specific musical features (though this has been done for films; [43]). However, indirect evidence focusing on physiological responses to music such as chills or tears, symptoms that are also strongly associated with feeling moved, has been provided. Sloboda [46] found that tears were most often associated with appoggiaturas, while shivers or chills were linked to excerpts containing new or surprising harmonies (see also [47]). Later and Panksepp [42] identified solo instruments that emerge from an orchestral background and crescendos as elicitors of chills. Such dynamic changes in loudness have also been found to be an indicator of chills in more recent research [40, 48, 49]. Relatedly, an experimental study found that increased loudness (acoustic intensity) and reduced brightness (proportion of high to low frequencies) resulted in more frequent reports of chills [39]. Further research has also identified the entrance of a voice and surprising changes or violations of expectations as structural elicitors of chills [47]. As Grewe et al. [40, 47] noted, there seems to be no consistent evidence for a specific chill-inducing acoustical pattern. Rather, interactions among different musical features might be more successful in eliciting a chill response, possibly also depending on additional psychological variables at the individual level such as empathy or familiarity. Theories of feeling moved Building on anthropological and ethnographic work, Fiske et al. [12] introduced kama muta as a universal tendency to respond emotionally (in a way that is often described as moved or touched) when communal sharing suddenly intensifies. Communal sharing is one of four basic schemes (or models) of relating to others focusing on what we have in common. It is expressed by giving according to need and ability (while the other three models are expressed by giving according to hierarchy, equality and proportionality, respectively), by bodily proximity, touch, synchrony, or food sharing (see [13], for a more detailed introduction of communal sharing and the remaining models). States of kama muta are thus characterized by appraisals of increased interpersonal closeness or connectedness (reflecting the intensification of communal sharing), by labeling them as moved, touched or heartwarming, as well as by sensations of tears, warmth in the chest and chills, by experiencing them as positive, and by renewed devotion to one’s relationships characterized by communal sharing [14, 15]. Empirical evidence has supported kama muta theory across several different cultural contexts [5, 16]. However, so far the empirical studies testing kama muta theory have only used stories and videos to evoke kama muta through the intensification of communal sharing between human characters. Fiske [14] assumes that the same appraisal of increased closeness or connectedness also accounts for instances where people describe being moved by music: “Sometimes the CS that suddenly intensifies is between musicians and audience, sometimes among the musicians or singers, sometimes among the audience, sometimes with the composer or even with the music itself.” However, so far, this proposition has not been tested. Kama muta theory defines the emotion of kama muta through the main appraisal theme, intensification of communal sharing (labelled an elicitor-specific eudaimonic emotion in recent theorizing, [17]). The labels that persons give their emotional experience, such as moved, touched, heartwarming, raptured and others, in English, are but one, though for diagnostic purposes often very important, index for the emotion [12]. Conversely, other theories define the emotion through the label “being moved”, encompassing assumed elicitors, subjective feelings, physiological signs and sensations and often also motivational tendencies (a feeling-specific eudaimonic emotion according to Landmann [17], see [6], for a review). Of these, two are particularly relevant for the present investigation. Konečni [18, 19] treats being moved as part of the aesthetic trinity, next to thrills or chills and states of aesthetic awe. Thereby, being moved represents strong emotional states that are experienced in response to the sublime (including musical stimuli). He contends that chills are, in the context of music, far more frequent and predictable (because they are shallower) than the state of being moved. He considers being moved to be a rarer response and determined by the personal associative context of the person. If this was the case, there should be low interindividual agreement on what the moving segments in a musical piece are. Conversely, kama muta theory suggests that being moved can be evoked reliably by musical passages because the music itself conveys an intensification of communal sharing. We also expect that for moving music, the occurrence of chills and feeling moved or touched coincide to a large degree. The other theory we draw on for the present investigation is the distancing-embracing model [20], which posits that being moved is involved in transforming negative states into pleasurable experiences in aesthetic perceptions, including music listening. Being moved is thereby conceptualized as a mixed emotion that is experienced as predominantly positive. The authors distinguish between two prototypes of being moved: a sadly moving and a joyfully moving variant [21, 22]. They expect that being moved plays a more important role in the enjoyment of sad stimuli. Indeed, empirical research has indicated that being moved is a mediating factor in explaining the enjoyment of artworks that are experienced as sad [22, 23]. Similar findings have also been obtained for sad music, and empathic responses were identified as a possible driver [7, 24]. Empathy, in turn, has been identified as a characteristic of at least an important variant of being moved, and empathic predispositions predict the propensity to feel moved [6, 21, 25]. It remains to be shown, however, whether feeling moved or touched by predominantly sad versus joyful pieces of music have the same profile, which would strengthen the argument that it is the same emotion in different affective contexts. To summarize, based on kama muta theory we predict that feeling moved or touched by music co-occurs with a sense of connectedness and weeping, feelings of warmth in the chest, and chills. While these attributes are compatible with all theories on the state of feeling moved (for overviews, see [6, 26]), the aesthetic trinity theory [18] predicts low agreement regarding which segments move individuals, while kama muta theory predicts high agreement for music that is preselected to be experienced as moving. This high agreement should be caused by similar emotional responses to musical features, and we shall attempt to identify such features. Lastly, we predict that feeling moved by sadly and joyfully moving music will show comparable correlation patterns. This would suggest that it is the same emotion in both cases, whether two variants of the same mixed emotion [21] or the same emotion with different concurrent emotions of sadness and joy [14]. Feeling moved as a response to music As briefly alluded to earlier, feeling moved has received increased theoretical and empirical attention in the context of music. In a questionnaire study a total of 141 participants indicated that feeling moved was their fourth most common emotional response in listening to music [1], and interviews reveal that individuals often say that they are moved in the context of strong and profound experiences with music [4]. Similarly, Scherer and Zentner [27] identified feeling moved as a common response when listening to music, involving symptoms such as moist eyes or chills (although they treated feeling moved as a vague emotional category) and a recent survey showed that feeling moved is the most commonly reported emotional state when investigating crying in response to music [28]. Items assessing feeling moved have been included in the Geneva Emotional Music Scale [29] and a hierarchical clustering approach suggests that feeling moved (as measured by the items moved and touched) form a distinct cluster of musically induced emotions [30], being most similar to the states of wonder and transcendence. In a recent review of factors affecting the enjoyment of sad music, Eerola et al. [24] identified feeling moved as a possible mediator, next to empathy or social surrogacy. Across two experiments Vuoskoski and Eerola [7] examined this proposition empirically. Testing 327 participants across two experiments, the authors presented several musical excerpts that differed in terms of movingness and sadness, and collected responses on felt sadness, feeling moved and liking. In the first experiment, feeling moved fully mediated the relationship between felt sadness and liking (see [23] for similar findings using film stimuli), and the second experiment further confirmed that this relationship could not be explained by the perceived beauty of the musical pieces. Rather, movingness appeared to contribute to the perceived beauty of sad music. Given these theoretical and empirical findings, Zickfeld [31] suggested that kama muta theory might incorporate and explain several factors contributing to the enjoyment of sad music (identified in Eerola et al.’s [24] review): feeling moved, empathy, social surrogacy, or more precisely communal sharing and affiliation, and experiencing sad music as positive and pleasurable. While it is easy to argue that music may increase affiliation through lyrics expressing tenderness and love [12], the ways in which instrumental music is to convey affiliation seem less straightforward. On one hand, a listener might experience an increase in affiliation by identifying or feeling a connection to the music in general (for example due to rhythmic entrainment; see e.g., [32]), the composer, the musicians, or other fans/listeners. Listening to familiar music may also remind us of nostalgic relationships with significant others: Konečni [18] argues that feeling moved by music is individually determined by the associative context or web that one has constructed with the musical piece, possibly also involving an affiliation towards certain aspects of the music. Evidence from an empirical study on nostalgia responses to music supports this assertion [33]. The authors found that musical pieces were rated as more nostalgic if they were perceived as familiar or more autobiographically salient. In addition, there is evidence that listeners can infer affiliative motives from musical improvisations [8], can feel connected to music through a form of empathy [34], and that listening to moving music from a specific culture can increase affiliation towards that culture [35]. On the other hand, in more interactive contexts feeling moved might occur due to concert attendants dancing or moving in synchrony, singing in unison, as well as perceiving the musicians performing music in synchrony. All of these aspects have been shown to elicit social bonding (e.g., [36]), and are part of communal sharing relationships that might intensify in specific contexts [13]. Musical chills. Further, strong emotional responses to music have been associated with the occurrence of chills, goosebumps or frissons [37–42]. Chills are considered a relatively common, positive psychophysiological response to music or aesthetic stimuli (see [37], for a review). Research has sometimes distinguished between chills as a subjective emotional response and goosebumps or piloerection as an objective physiological response [43]. In the present manuscript, we follow Maruskin et al. [44], who argued that goosebumps represent one physiological component of pleasurable chills. Empirical studies have linked the occurrence of chills to strong experiences of feeling moved using qualitative methods, self-report ratings and more observational techniques such as camera devices [5, 22, 38, 43, 45]. Bannister [39] found in a comprehensive survey on chills responses to music that a high number of participants highlighted social aspects such as feelings of connectedness, the human voice, and perceived relationships between virtual agents (i.e., musical instruments/parts) as evoking chills. Similarly, a recent survey study identified moving chills as one of three types of chill responses to aesthetic stimuli, which co-occurred with feeling moved, tenderness, and tears (the other types being warm chills and cold chills, Bannister [38]; see also [44]). Such results fit previous findings and theories on feeling moved and highlight the importance of chills as a correlate of feeling moved or touched in response to music. Musical chills. Further, strong emotional responses to music have been associated with the occurrence of chills, goosebumps or frissons [37–42]. Chills are considered a relatively common, positive psychophysiological response to music or aesthetic stimuli (see [37], for a review). Research has sometimes distinguished between chills as a subjective emotional response and goosebumps or piloerection as an objective physiological response [43]. In the present manuscript, we follow Maruskin et al. [44], who argued that goosebumps represent one physiological component of pleasurable chills. Empirical studies have linked the occurrence of chills to strong experiences of feeling moved using qualitative methods, self-report ratings and more observational techniques such as camera devices [5, 22, 38, 43, 45]. Bannister [39] found in a comprehensive survey on chills responses to music that a high number of participants highlighted social aspects such as feelings of connectedness, the human voice, and perceived relationships between virtual agents (i.e., musical instruments/parts) as evoking chills. Similarly, a recent survey study identified moving chills as one of three types of chill responses to aesthetic stimuli, which co-occurred with feeling moved, tenderness, and tears (the other types being warm chills and cold chills, Bannister [38]; see also [44]). Such results fit previous findings and theories on feeling moved and highlight the importance of chills as a correlate of feeling moved or touched in response to music. Feeling moved and musical features To the best of our knowledge, there has been no direct research linking responses of feeling moved by music to specific musical features (though this has been done for films; [43]). However, indirect evidence focusing on physiological responses to music such as chills or tears, symptoms that are also strongly associated with feeling moved, has been provided. Sloboda [46] found that tears were most often associated with appoggiaturas, while shivers or chills were linked to excerpts containing new or surprising harmonies (see also [47]). Later and Panksepp [42] identified solo instruments that emerge from an orchestral background and crescendos as elicitors of chills. Such dynamic changes in loudness have also been found to be an indicator of chills in more recent research [40, 48, 49]. Relatedly, an experimental study found that increased loudness (acoustic intensity) and reduced brightness (proportion of high to low frequencies) resulted in more frequent reports of chills [39]. Further research has also identified the entrance of a voice and surprising changes or violations of expectations as structural elicitors of chills [47]. As Grewe et al. [40, 47] noted, there seems to be no consistent evidence for a specific chill-inducing acoustical pattern. Rather, interactions among different musical features might be more successful in eliciting a chill response, possibly also depending on additional psychological variables at the individual level such as empathy or familiarity. The present study Not only music performance but also music perception seems to be inherently social. Listeners can infer affiliative motives from musical improvisations [8] and feel connected to music through a form of empathy [34], and music can increase connectedness towards aspects conveyed and embodied by music [35]. While there exists ample evidence that feeling moved often constitutes strong emotional responses to musical stimuli [1, 4], the underlying mechanisms or conditions that moderate this reaction are yet to be investigated. As states of feeling moved typically occur in response to significant social or communal events, we argue that the inherent affiliative signals in music can evoke this particular emotion. Listeners can feel connected to aspects of the music, aspects of the performer(s), aspects of co-listeners, or an interaction of these variables [31]. In the present study we aim to investigate experiences labeled as feeling moved in response to music using continuous self-reports, a common paradigm in research on music and emotions that allows to model listeners’ responses to music dynamically (see [11, 50, 51]). Instead of prompting participants to rate their emotional reactions after being exposed to a stimulus, continuous self-report paradigms collect responses while participants are presented with a stimulus. This methodology has two obvious strengths compared to self-report ratings after stimulus presentation, and extends and goes beyond previous studies on the role of feeling moved in response to music [7, 52]. First, it allows us to obtain a more nuanced picture of the psychological dynamics and mechanisms unfolding while listening to music. It enables us to test whether the emotion components of feeling moved that have been found to co-occur in other contexts [6] also apply to music listening. For example, previous research has found a positive correlation between feeling sad and feeling moved by music [7]. However, this relationship lacks time specificity: It is not known whether some parts of the music are perceived as sad and others as moving, or whether the same parts are perceived as sad and moving at the same time. Second, it also allows us to test how experiences of feeling moved unfold based on specific musical features. As specified earlier, there is no direct evidence with regard to what musical or psychoacoustic features are associated with feeling moved. In the present project we employed a continuous self-report paradigm based on the set-up of a recent empirical study exploring continuous ratings of different aspects of feeling moved in response to videos [53]. Participants were presented with a Likert-type scale and asked to change their ratings when their experiences and perceptions changed while listening to the musical excerpts. Seven variables or ratings were assessed continuously for all excerpts (although each participant rated only one variable per excerpt) to target the feeling, physiological, and appraisal components of the state of the emotion. To assess the feeling component, we included an item of feeling moved or touched as used in a previous study [53]. For the physiological component, we included items on chills and feelings of warmth (in the chest) (based on Schubert et al. [53]). The quality of warmth in the center of the chest has been strongly associated with feeling moved (e.g., [5, 53]), while there has been only limited research exploring the reaction in response to musical stimuli (e.g., [38]). At the moment it is not clear whether these feelings of warmth in the chest are associated with an actual objective temperature increase, or whether participants embody the metaphor of communality and feel social warmth (see [6, 54, 55]). Note that we did not assess the common response of tears or moist eyes continuously (this was rated only after the music excerpt had ended), since our pilot study indicated a rather low occurrence of self-reported tears or moist eyes in response to our stimuli. For the appraisal component, we wanted to assess the affiliative signals or communal sharing aspects conveyed by the music. While previous studies have measured this aspect using either items with regard to how close someone feels to another target [5], how close the protagonists are to each other, or with the inclusion-of-the-other-in-the-self scale measuring closeness by showing two circles increasing in overlap [53, 56], we felt that it might seem artificial to probe how close someone feels to a certain musical piece while it unfolds. Based on the findings of a pilot study, we decided to assess the extent to which the participants felt a sense of connection when listening to music, relating to findings that individuals are able to detect affiliative intentions in instrumental musical stimuli [8]. Given that moving music is perceived as more beautiful than non-moving music [7, 28], we also included an item on perceived beauty in order to test whether continuous ratings of beauty are similar to ratings of feeling moved. Finally, in line with previous studies, we also assessed how sad and joyful participants perceived the music. Specifically, we targeted the emotions perceived in the music and not felt emotions for these two items, although it should be noted that the two processes are strongly related (see [7]). This choice was made due to the possibility that the continuous ratings of experienced joy could be confounded by enjoyment of the music as a result of being moved by the music. We were, however, more interested in the elicitors of feeling moved or touched. Some theorizations have distinguished between the categories of being sadly and joyfully moved [21], and have found different predictions for a number of variables (e.g., [22]). Although all theories on feeling moved emphasize that the emotion is experienced as primarily positive, they do not exclude the possibility that it occurs together at the same time with negative emotions such as sadness. Thus, in the present study we focused on including musical excerpts that were rated as both moving and sad, as well as both moving and joyful in order to explore possible diverging patterns. Based on previous findings and theories, specifically kama muta theory, we derived a number of different predictions that were pre-registered prior to conducting the study (https://osf.io/76adr). We predicted that the time course of: feeling moved or touched (feeling component) correlates positively with the time course of experiences of warmth in the chest, experienced chills (physiological component), and experiencing a sense of connection (appraisal component) across all excerpts; feeling moved correlates positively with the time course of perceived beauty across all musical pieces; feeling moved correlates positively with the time course of perceiving the music as sad for sadly moving excerpts, while it does less so for joyfully moving excerpts; feeling moved correlates positively with the time course of perceiving the music as joyful for joyfully moving excerpts, while it does less so for sadly moving excerpts. In addition, we explored relations between feeling moved and musical or acoustic features of the music. As there exists only indirect evidence based on musical chills, we did not pre-register any specific hypotheses but rather treated this aspect of the study as exploratory. Finally, we explored the relationship between ratings of feeling moved and two facets of trait empathy [57]; empathic concern and fantasy. Fantasy denotes the tendency to imaginatively transpose oneself into the feelings and experiences of fictitious characters, and it has been found to predict feeling moved by sad music in prior studies [7, 52]. Empathic concern denotes the sympathy and compassion one feels for others in need. It has been found to predict feeling moved across several studies and cultures and 3000+ participants with an overall effect size of r = .35 [5, 58]. Materials and methods The study was approved by the internal ethics committee of the Department of Psychology at the University of Oslo. In addition, we pre-registered the main parts of the study (https://osf.io/76adr/). Deviations from the pre-registration protocol are denoted explicitly as such. All data, analysis scripts and materials are available at our project page (https://osf.io/xgj85/). Participants Previous studies employing continuous self-report paradigms have not considered sample size justifications systematically [11, 50, 51]. As an exception, a recent methodological paper considered between 20 and 30 raters per cell (combination of independent variables) to be sufficient, though this number was dependent on the specific study design [59]. We originally registered to collect 40–50 participants per cell based on a previous study using a similar procedure [53]. Performing a post-hoc precision analysis on our data following McKeown and Sneddon [59] suggested that this number was sufficient (S5 Fig in S1 File). As we had 49 cells in total (7 excerpts x 7 ratings) and each participant completed seven of these, we aimed for at least 350 participants. Due to exclusions five of the 49 cells had fewer than 40 participants in the final dataset (see S4 Table in S1 File). In total, 423 participants were recruited on Amazon MTurk, requesting only workers with at least 95% approval rating for tasks on Amazon MTurk and location set to the US. All participants were asked to provide informed consent before participating. After applying the pre-registered exclusion criteria, the final sample consisted of 415 US American participants (197 women, 183 men, 35 unspecified gender) ranging from 19 to 68 years of age (Mage = 36.05, SDage = 10.52). Responses to an excerpt were excluded if participants spent less than two minutes on the page presenting the excerpt, which was recorded with a timer, thus resulting in some participants completing less than seven excerpts and ratings. In addition, participants were excluded if they indicated an age younger than 18, a different nationality than the US, if more than 50% of the questions were unanswered, or if they failed a probe item that assessed whether they understood the instructions. We excluded non-US participants to ensure that the participants had a shared, consistent understanding of the emotional labels and concepts used in the rating tasks, and to control for possible effects of cultural background on emotional experiences in response to music (cf. [60]). Procedure The present experiment followed a 7 (within: musical excerpt) x 7 (within: rating scale) mixed design. Each of the seven excerpts was randomly coupled with one of the seven rating scales for each participant. More specifically, for the first excerpt and rating, participants were randomly presented with one out of 7x7 possible combinations. For the second excerpt, participants were randomly presented with one out of 6x6 possible combinations and so on. The type of combination and order of all excerpts and ratings was randomized for each participant individually. Importantly, each excerpt and rating scale was only presented or used once by each participant. The general continuous self-report paradigm was based on Schubert et al. [53]. The study was created and carried out using the Qualtrics online platform. After providing informed consent, participants were presented with detailed instructions about their task and were familiarized with the continuous rating scales. They were then presented with the seven different combinations of musical excerpt and rating scale in individual random order. Before each excerpt, they were informed what aspect they should rate and reminded about how to do that. Excerpts played automatically and participants were instructed to rate their experiences or perceptions continuously, changing their rating when their experience or perception changed. For that purpose, participants were shown a Likert-type rating scale while listening to the excerpt. Each scale included five scale points ranging from ‘’Not at all (1)’, ’A little (2)’, ’Moderately (3)’, ’Very (4)’, to ’Extremely (5)’ (with numbers in brackets referring to the scale point or the specific number key; see below for a specific example). Participants were instructed to update their response by either clicking on an option with their mouse, using the arrow keys to decrease or increase their rating, or using the number keys on their keyboard to indicate a response between 1 and 5. The lowest scale point was selected as the default when participants started a new excerpt (an example of the rating screen is provided in the S1 File). We retained the exact time at which participants changed their rating and matched it to the time code of the musical excerpts using a JavaScript code. Musical excerpts were hosted on YouTube and embedded in the survey by hiding player controls and visuals. Materials Each participant was presented with a total of seven musical excerpts. These excerpts were selected based on a pilot study (see S1 File and https://osf.io/y4dfa/), and a subset of them have been utilized in previous studies investigating the emotional impact of music [7, 52, 61]. An excerpt of two to three minutes was taken from each piece, the exact length of the excerpt depending on the phrase structure of the piece in question. Based on ratings of the pilot study, the excerpts were grouped into sadly moving (perceived high in sadness and feeling moved, but low in happiness), and joyfully moving (perceived high in happiness and feeling moved, but low in sadness). Three excerpts were included in the sadly moving category (‘Allegri’, ‘Olafur’, and ‘Oblivion’), and four in the joyfully moving category (‘Band of Brothers’, ‘Hoppipolla’, ‘Vltava’, and ‘Explosions’). Originally, the intention was to have three excerpts each in the sadly and joyfully moving categories, as well as one ‘neutral’ excerpt that was perceived to the same degree as joyful and sad. However, due to a coding error in the numbering of the pilot stimuli, ‘Explosions’ was mistakenly identified as the neutral excerpt. As a result, no neutral excerpts were included in the main experiment. More details about this, as well as a detailed overview of each excerpt including duration is provided in the S1 File and on https://osf.io/xgj85/. Participants were presented with seven different scales: ‘Perceived Sadness’, ‘Perceived Joy’, ‘Feeling Moved or Touched’, ‘Sense of Connection’, ‘Perceived Beauty’, ‘Warmth (in the chest)’, and ‘Chills’. Continuous rating scales targeted perceived sadness (“Right now, how sad does the music sound?”), perceived joy (“Right now, how joyful does the music sound?”), perceived beauty (“Right now, how beautiful does the music sound?”), feeling moved or touched (“Right now, how moved or touched do you feel?”), felt sense of connection (“Right now, to what degree do you feel a sense of connection?”), felt warm feeling in the chest (“Right now, to what degree do you experience a warm feeling in the chest?”), and felt chills or goosebumps (“Right now, to what degree do you experience chills (goosebumps)?”). All ratings were completed on a 5-point scale ranging from ’Not at all (1)’, ’A little (2)’, ’Moderately (3)’, ’Very (4)’, to ’Extremely (5)’ (e.g., for the sadness rating: Not at all sad (1), A little sad (2), Moderately sad (3), Very sad (4), Extremely sad (5)). Note, that previous studies have typically assessed chills using a dichotomous measure probing whether participants experienced chills or not [37]. Based on Schubert et al. [53], we were interested in differentiating various intensities of chill responses. After listening and continuously rating each excerpt, participants also completed additional measures pertaining to each of the excerpts. First, they were asked to indicate whether they had heard the piece before (with answer options: Definitely yes, probably yes, might or might not, probably not, definitely not). Then, they completed a 7-point scale asking about how much they had enjoyed the excerpt, anchored at ‘Not at all (0)’ and ‘Very Much (6)’. We also included a self-report item asking whether participants had experienced tears or moist eyes while listening to the excerpt on the same 7-point scale. Finally, participants were presented with a dichotomous item asking about technical difficulties during playback. Upon completing the ratings for all seven excerpts, participants were asked to complete the empathic concern (Chronbach’s ɑ = .91) and fantasy (ɑ = .85) subscales of the Interpersonal Reactivity Index (IRI; [57]), a questionnaire designed to assess interindividual differences in empathic responding. Each subscale consisted of seven items answered on a 5-point scale ranging from ‘Does not describe me well’ to ‘Describes me very well’. Participants also completed demographic information (including gender, age, nationality, number of children, whether they had a pet, and a question about relationship status). We also included an item asking whether participants understood any of the lyrics (two of the excerpts included lyrics; Allegri in Latin, and Hoppipolla in Icelandic/Hopelandic) with the answer options ’Yes’ and ’No’. Finally, we included an item to assess musical proficiency, asking whether participants play/have played any musical instruments, and if yes, for how long. Participants Previous studies employing continuous self-report paradigms have not considered sample size justifications systematically [11, 50, 51]. As an exception, a recent methodological paper considered between 20 and 30 raters per cell (combination of independent variables) to be sufficient, though this number was dependent on the specific study design [59]. We originally registered to collect 40–50 participants per cell based on a previous study using a similar procedure [53]. Performing a post-hoc precision analysis on our data following McKeown and Sneddon [59] suggested that this number was sufficient (S5 Fig in S1 File). As we had 49 cells in total (7 excerpts x 7 ratings) and each participant completed seven of these, we aimed for at least 350 participants. Due to exclusions five of the 49 cells had fewer than 40 participants in the final dataset (see S4 Table in S1 File). In total, 423 participants were recruited on Amazon MTurk, requesting only workers with at least 95% approval rating for tasks on Amazon MTurk and location set to the US. All participants were asked to provide informed consent before participating. After applying the pre-registered exclusion criteria, the final sample consisted of 415 US American participants (197 women, 183 men, 35 unspecified gender) ranging from 19 to 68 years of age (Mage = 36.05, SDage = 10.52). Responses to an excerpt were excluded if participants spent less than two minutes on the page presenting the excerpt, which was recorded with a timer, thus resulting in some participants completing less than seven excerpts and ratings. In addition, participants were excluded if they indicated an age younger than 18, a different nationality than the US, if more than 50% of the questions were unanswered, or if they failed a probe item that assessed whether they understood the instructions. We excluded non-US participants to ensure that the participants had a shared, consistent understanding of the emotional labels and concepts used in the rating tasks, and to control for possible effects of cultural background on emotional experiences in response to music (cf. [60]). Procedure The present experiment followed a 7 (within: musical excerpt) x 7 (within: rating scale) mixed design. Each of the seven excerpts was randomly coupled with one of the seven rating scales for each participant. More specifically, for the first excerpt and rating, participants were randomly presented with one out of 7x7 possible combinations. For the second excerpt, participants were randomly presented with one out of 6x6 possible combinations and so on. The type of combination and order of all excerpts and ratings was randomized for each participant individually. Importantly, each excerpt and rating scale was only presented or used once by each participant. The general continuous self-report paradigm was based on Schubert et al. [53]. The study was created and carried out using the Qualtrics online platform. After providing informed consent, participants were presented with detailed instructions about their task and were familiarized with the continuous rating scales. They were then presented with the seven different combinations of musical excerpt and rating scale in individual random order. Before each excerpt, they were informed what aspect they should rate and reminded about how to do that. Excerpts played automatically and participants were instructed to rate their experiences or perceptions continuously, changing their rating when their experience or perception changed. For that purpose, participants were shown a Likert-type rating scale while listening to the excerpt. Each scale included five scale points ranging from ‘’Not at all (1)’, ’A little (2)’, ’Moderately (3)’, ’Very (4)’, to ’Extremely (5)’ (with numbers in brackets referring to the scale point or the specific number key; see below for a specific example). Participants were instructed to update their response by either clicking on an option with their mouse, using the arrow keys to decrease or increase their rating, or using the number keys on their keyboard to indicate a response between 1 and 5. The lowest scale point was selected as the default when participants started a new excerpt (an example of the rating screen is provided in the S1 File). We retained the exact time at which participants changed their rating and matched it to the time code of the musical excerpts using a JavaScript code. Musical excerpts were hosted on YouTube and embedded in the survey by hiding player controls and visuals. Materials Each participant was presented with a total of seven musical excerpts. These excerpts were selected based on a pilot study (see S1 File and https://osf.io/y4dfa/), and a subset of them have been utilized in previous studies investigating the emotional impact of music [7, 52, 61]. An excerpt of two to three minutes was taken from each piece, the exact length of the excerpt depending on the phrase structure of the piece in question. Based on ratings of the pilot study, the excerpts were grouped into sadly moving (perceived high in sadness and feeling moved, but low in happiness), and joyfully moving (perceived high in happiness and feeling moved, but low in sadness). Three excerpts were included in the sadly moving category (‘Allegri’, ‘Olafur’, and ‘Oblivion’), and four in the joyfully moving category (‘Band of Brothers’, ‘Hoppipolla’, ‘Vltava’, and ‘Explosions’). Originally, the intention was to have three excerpts each in the sadly and joyfully moving categories, as well as one ‘neutral’ excerpt that was perceived to the same degree as joyful and sad. However, due to a coding error in the numbering of the pilot stimuli, ‘Explosions’ was mistakenly identified as the neutral excerpt. As a result, no neutral excerpts were included in the main experiment. More details about this, as well as a detailed overview of each excerpt including duration is provided in the S1 File and on https://osf.io/xgj85/. Participants were presented with seven different scales: ‘Perceived Sadness’, ‘Perceived Joy’, ‘Feeling Moved or Touched’, ‘Sense of Connection’, ‘Perceived Beauty’, ‘Warmth (in the chest)’, and ‘Chills’. Continuous rating scales targeted perceived sadness (“Right now, how sad does the music sound?”), perceived joy (“Right now, how joyful does the music sound?”), perceived beauty (“Right now, how beautiful does the music sound?”), feeling moved or touched (“Right now, how moved or touched do you feel?”), felt sense of connection (“Right now, to what degree do you feel a sense of connection?”), felt warm feeling in the chest (“Right now, to what degree do you experience a warm feeling in the chest?”), and felt chills or goosebumps (“Right now, to what degree do you experience chills (goosebumps)?”). All ratings were completed on a 5-point scale ranging from ’Not at all (1)’, ’A little (2)’, ’Moderately (3)’, ’Very (4)’, to ’Extremely (5)’ (e.g., for the sadness rating: Not at all sad (1), A little sad (2), Moderately sad (3), Very sad (4), Extremely sad (5)). Note, that previous studies have typically assessed chills using a dichotomous measure probing whether participants experienced chills or not [37]. Based on Schubert et al. [53], we were interested in differentiating various intensities of chill responses. After listening and continuously rating each excerpt, participants also completed additional measures pertaining to each of the excerpts. First, they were asked to indicate whether they had heard the piece before (with answer options: Definitely yes, probably yes, might or might not, probably not, definitely not). Then, they completed a 7-point scale asking about how much they had enjoyed the excerpt, anchored at ‘Not at all (0)’ and ‘Very Much (6)’. We also included a self-report item asking whether participants had experienced tears or moist eyes while listening to the excerpt on the same 7-point scale. Finally, participants were presented with a dichotomous item asking about technical difficulties during playback. Upon completing the ratings for all seven excerpts, participants were asked to complete the empathic concern (Chronbach’s ɑ = .91) and fantasy (ɑ = .85) subscales of the Interpersonal Reactivity Index (IRI; [57]), a questionnaire designed to assess interindividual differences in empathic responding. Each subscale consisted of seven items answered on a 5-point scale ranging from ‘Does not describe me well’ to ‘Describes me very well’. Participants also completed demographic information (including gender, age, nationality, number of children, whether they had a pet, and a question about relationship status). We also included an item asking whether participants understood any of the lyrics (two of the excerpts included lyrics; Allegri in Latin, and Hoppipolla in Icelandic/Hopelandic) with the answer options ’Yes’ and ’No’. Finally, we included an item to assess musical proficiency, asking whether participants play/have played any musical instruments, and if yes, for how long. Results Data preparation An overview of the general data preparation process is provided in Fig 1. First, for each participants’ timestamped ratings we created time series at 1Hz resolution (one rating per second; Fig 1A). For example, if a participant started with a rating of 1 on a specific scale and changed their rating to a 3 at 30 s, the time series would show a rating of 1 from 0 s to 29 s, and a rating of 3 from 30 s until the second the rating was changed again (or until the end of the musical excerpt if the rating did not change). Second, we averaged the time series for each scale-excerpt combination across participants, resulting in seven averaged time series ratings for each of the seven excerpts (Fig 1B). Consistency of time series was computed with intraclass correlations, which represent a measure of inter-rater agreement (ICC; [62]). The mean ICC across all excerpt-rating combinations was .86 and all indices except the one for perceived sadness and Hoppipolla were above .60 (see S4 Table in S1 File), indicating good agreement among raters. Third, we decreased the resolution of the time scales by aggregating judgments within units of three consecutive seconds. We assumed that participants were not able to report their momentary experiences within the exact same second they appeared. Previous research has suggested that 3 seconds represent the temporal building blocks of many aspects related to human perceptual or motor abilities [63]. Therefore, we registered three seconds as an adequate time bin (Fig 1C). To test the robustness of our aggregation in the light of findings suggesting that aesthetic ratings are stable at 1s responses [64], we repeated the main cross-correlations for the 1Hz data. Effects were similar, though a bit stronger. An overview is provided in the (S8 Table in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Overview of the data handling process. (A) We recorded individual time series for each scale and musical excerpt. (B) Individual time series were averaged across each scale and musical excerpt combination (error bars represent 95% CIs). (C) The time series was averaged within units of three consecutive seconds. (D) We applied a cubic spline interpolation using six sampling points in order to remove trends in the time series. The depicted example graphs represent the feeling moved ratings for Explosions as an illustration of the overall process. https://doi.org/10.1371/journal.pone.0261151.g001 Fourth, we detrended the time series. Most time series are non-stationary, meaning that they show linear or higher-order increases or decreases. Comparing two non-stationary time series would inflate correlations, as they tend to cross-correlate simply due to the fact that they have trends in common. Detrending, removing the trend in time series, is one possible way to achieve the stationarity of time series data [65, 66]. Inspecting our time series, all averaged curves showed increases in ratings over time and some also decreases in ratings towards the end. More specifically, they indicated positive linear and negative quadratic trends. We detrended all time series using a cubic spline interpolation [59, 65]. Splines can be considered as an extension of polynomial regression that divides the time series into a number of k intervals, delimited by knots. Employing a cubic spline, a regression with three parameters (linear, quadratic, and cubic) is fit at each knot. We used five knots and therefore six intervals for all time series ([67]; Fig 1D). We employed two additional detrending techniques: the residual and difference methods. For the residual method we removed the linear and quadratic trends from all curves by regressing them separately in a multiple regression on an index of time in seconds and its square, and saved the unstandardized residuals (Shumway and Stoffer [66]; S5 Table in S1 File). In differencing, the difference between the current observation and the previous observation is calculated for the whole time series. The direction of all main effects was similar with the three detrending methods. However, while the differencing and spline methods produced rather similar effects, the residual method showed considerably stronger effects for most measures (see S6 Table in S1 File). Note, that we originally registered to employ the residual detrending method. However, cubic splines have been considered to be superior when it comes to removing slow drifts and have been recommended to handle time series data when collecting emotional responses [59, 68]. Results of the two additional detrending methods are reported in the S1 File. We also report the results on the original un-detrended data. While these effects showed similar direction as the detrended effects, they were much stronger, which seems to be based on the underlying trends inflating and overestimating cross-correlations. The phenomenon of feeling moved tends to happen over shorter-time intervals and is not expected to demonstrate long-term trajectories, as typically assumed for emotional experiences (in contrast to longer-lasting moods; Beedie et al. [69]). Hence for the purpose of this study detrending does not affect this significantly. Especially, the cubic spline detrending that we employ ensures that we capture these shorter-term fluctuations while removing the long-term trajectories which are more an artefact resulting from the non-stationary nature of the emotion ratings (e.g., [70]). After inspecting the time-series, we noticed that most time-series increased sharply for the first few seconds of the excerpts. Such early changes likely reflect the change from silence to music (rather than changes within the music) and have been called initial orientation time, the time it takes for continuous ratings to fall within some specific range [71]. In a previous study the median initial orientation time was observed at 8s [71]. As such sharp increases that might not reflect changes related to the music could inflate relationships across the time-series, we excluded the first three time-bins (i.e., the first nine seconds) of every excerpt. Note that this decision was not pre-registered. Acoustic feature extraction Eleven acoustic features were chosen to broadly capture the loudness-related, timbral, tonal and rhythmic aspects of the musical excerpts (see Table 1 for a description of each acoustic feature). The features can typically be classified into two categories based on the duration of the analysis-window used during the feature extraction process [72, 73]. Short-term features were extracted using a 50ms window with a 50% overlap and encompassed loudness and timbral properties. While loudness was captured by the feature root-mean-square energy, timbral features primarily comprised spectral shape descriptors, namely spectral centroid, spectral spread, spectral roll-off, entropy, roughness, and flatness. In addition, spectrotemporal fluctuations were captured by spectral flux and temporal fluctuations by zero-crossing rate. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Overview and description of acoustic features extracted. https://doi.org/10.1371/journal.pone.0261151.t001 Long-term features capture context-dependent tonal and rhythmic aspects of music and are extracted using a three second window with an overlap of 67%. Tonal variation was captured by key strength or key clarity while rhythmic variation was captured by pulse clarity. All features were extracted from each excerpt using the MIRtoolbox [78]. Subsequently, spline detrending was performed on the acoustic features, and the first 9 seconds of the excerpt were excluded from analyses as done for the emotional ratings. Finally, we created time bins of three seconds in order to match the self-report ratings. Detrending of the acoustic features was necessary to ensure comparability with the detrended self-report time series, which removed potential slow drift or long-term features. The transformation procedure was based on previous studies focusing on comparing acoustic features with neurophysiological responses [72]. Cross-correlations In order to compare changes in the different ratings over time, we computed cross-correlation functions (CCF) and their 95% confidence intervals. To calculate cross-correlations and estimate confidence intervals across sadly and joyfully moving excerpts, cross-correlations for each excerpt were employed in a random-effects meta-analysis using restricted maximum likelihood estimation in the metafor package in R [79]. Note that this approach was not pre-registered. Alternatively, we employed Fisher-Z transformations in order to obtain overall coefficients as specified in the pre-registration. Cross-correlations were first transformed using a Fisher-Z transformation, averaged, and finally back-transformed. Findings differed minimally and are presented in the (S7 Table in S1 File). We focused on cross-correlations at lag zero: A high cross-correlation between two scales at this lag means that both ratings change concurrently in the same direction as the music unfolds. Note, that a cross-correlation function at lag zero is the same as a zero-order Pearson correlation coefficient. For robustness, we also calculated the main analyses using Spearman correlation coefficients that are presented in the S9 Fig in S1 File. Similar to a previous study [53], we did not observe any systematic changes across other lags. An overview of the correlations between feeling moved or touched and all other continuous ratings, averaged for the different types of excerpts (after detrending employing the spline method) is presented in Fig 2. Additional results before detrending and after detrending employing the residual and difference methods are presented in S6 Table in S1 File. We also provide cross-correlations among the other variables in S9 Table in S1 File. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Cross-correlation functions (CCF) between feeling moved and touched and the six main variables across the musical excerpts. CCFs were calculated at lag 0 (meaning that both time series are compared at the same time) using cubic spline detrended timeseries. Sadly and joyfully moving songs are grouped separately. Overall estimates and confidence intervals are constructed employing a random-effects meta-analysis using restricted maximum likelihood estimation. Error bars represent 95% confidence intervals. RE = random effects. As heterogeneity measures we included Cochran’s Q and I2. https://doi.org/10.1371/journal.pone.0261151.g002 Warmth, chills, and a sense of connection. As predicted in H1, feeling moved or touched cross-correlated highly with experiencing warmth in the chest with an average of a cross-correlation function at lag 0 (CCF0) of .56 [.32, .80] across all excerpts. The effect was somewhat smaller for sadly than for joyfully moving excerpts (sadly: CCF0 = .50 [-.13, 1.13], joyfully: CCF0 = .61 [.47, .75]), which was based on one sadly moving excerpt showing a negative effect (Oblivion: CCF0 = -.17 [-.47, .13]). Similarly, feeling moved or touched cross-correlated highly with experiencing chills or goosebumps with an average of CCF0 = .63 [.51, .74] across all excerpts. The effect did only differ slightly between sadly (CCF0 = .65 [.45, .84]) and joyfully moving excerpts (CCF0 = .60 [.45, .76]). Finally, feeling moved or touched and feeling a sense of connectedness correlated highly with an average of CCF0 = .66 [.54, .76] across all excerpts. The effect was significantly stronger for joyfully (CCF0 = .75 [.63, .86]) than sadly moving excerpts (CCF0 = .51 [.39, .64]). Perceived beauty. As predicted in H2, feeling moved or touched cross-correlated highly with perceived beauty with an average of CCF0 = .60 [.49, .71] across all excerpts. The sadly moving excerpts (CCF0 = .58 [.32, .83]) showed similar effects compared to the joyfully moving excerpts (CCF0 = .60 [.47, .74]), though higher variation across excerpts. The averaged (non-detrended) continuous ratings of feeling moved or touched, perceived beauty, and feeling a sense of connection for the sadly and joyfully moving excerpts are displayed in Fig 3 and the detrended (cubic spline) ratings are presented in the S6 Fig in S1 File. The (non-detrended) ratings of feeling moved or touched, chills, and warmth in the chest are presented in Fig 4 (and the detrended ratings in S7 Fig in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. The averaged non-detrended continuous ratings of feeling moved or touched, perceived beauty, and feeling a sense of connection for the sadly (Upper Row) and joyfully moving (Lower Row) music excerpts. Ratings were provided on a 5-point scale (ranging from 1 (not at all) to 5 (extremely)) and aggregated in 3s time bins. https://doi.org/10.1371/journal.pone.0261151.g003 Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. The averaged (non-detrended) continuous ratings of feeling moved or touched, warmth in the chest, and chills for the sadly (Upper Row) and joyfully moving (Lower Row) music excerpts. Ratings were provided on a 5-point scale (ranging from 1 (not at all) to 5 (extremely)) and aggregated in 3s time bins. https://doi.org/10.1371/journal.pone.0261151.g004 Perceived sadness. The cross-correlation between feeling moved or touched and perceiving the excerpt as sad was CCF0 = .02 [-.34, .38] across all musical excerpts. As hypothesized in H3, the effect was much stronger for the sadly moving excerpts (CCF0 = .49 [.23, .76]) than for the joyfully moving excerpts (CCF0 = -.33 [-.59, -.07]), which showed an effect in the opposite direction. Perceived joy. The cross-correlation between feeling moved or touched and perceiving the excerpt as joyful was CCF0 = .61 [.48, .75] across all excerpts. Partly confirming predictions in H4, the effect was somewhat stronger for joyfully moving (CCF0 = .67 [.59, .74]) than sadly moving excerpts (CCF0 = .49 [.08, .90]), although this difference was driven by greater variation across the sadly moving excerpts. Except for one sadly moving excerpt (Oblivion: CCF0 = .05 [-.26, .35]), all musical excerpts showed a strong positive cross-correlation above .59. The averaged (non-detrended) continuous ratings of feeling moved or touched, perceived sadness, and perceived joy for the sadly and joyfully moving excerpts are displayed in Fig 5 and the detrended (cubic spline) ratings are presented in the (S8 Fig in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. The averaged (non-detrended) continuous ratings of feeling moved or touched, perceived sadness, and perceived joy for the sadly (Upper Row) and joyfully moving (Lower Row) music excerpts. Ratings were provided on a 5-point scale (ranging from 1 (not at all) to 5 (extremely)) and aggregated in 3s time bins. https://doi.org/10.1371/journal.pone.0261151.g005 Exploratory analyses The analyses reported in the following section were not pre-registered and/or did not test a pre-registered hypothesis, and should thus be considered exploratory. Enjoyment, familiarity and self-reported tears. In order to explore the relationship between liking, familiarity, and experiencing tears or moist eyes, liking of the musical excerpt was regressed on familiarity, tears or moist eyes and their interaction in a multilevel model. Intercepts were allowed to vary randomly according to participant and excerpt type. First, we found a main effect of familiarity. Higher familiarity with an excerpt increased the liking of that excerpt with an unstandardized coefficient of B = -.42 [-.53, -.32], t(2380) = -8.05, p < .001. In addition, we observed an interaction effect of familiarity and tears, B = .08 [.03, .12], t(2292) = 3.33, p < .001. Highly familiar excerpts were generally liked more. However, when combined with a strong response of tears, low familiarity resulted in more enjoyment than high familiarity and no or medium tear response. Acoustic and musical correlates of feeling moved. Spearman correlations were calculated between the detrended continuous ratings of feeling moved or touched and the eleven acoustic features (extracted using the MIRtoolbox). Fisher’s [80] Z transformation was then applied to the correlation values followed by adjusting by the factor 1 / √(df − 3), where df represents the estimated number of effective degrees of freedom calculated using the approach described by Pyper and Peterman [81]. The subsequent corrected z value was then converted to a respective p value. This procedure is the same as that carried out in previous studies focusing on the correlation between acoustic features and continuous neural responses [72]. The results of the correlation analyses are displayed in Fig 6. Joyfully moving excerpts exhibited similar positive correlation profiles between feeling moved or touched and the acoustic features representing loudness (rms), sensory dissonance (roughness) and spectrotemporal variations (flux). In addition, Vltava displayed a significant positive correlation with zero-crossing rate (zcr) and a negative correlation with key clarity, while Hoppipolla displayed a positive correlation with spectral rolloff. However, among the three sadly moving excerpts, only Allegri displayed significant correlations of feeling moved or touched with zero-crossing rate (zcr) and spectral entropy, with Oblivion displaying a similar correlation profile. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 6. A visualization of the spearman correlations between continuous ratings of feeling moved or touched, and the acoustic and musical features of the music excerpts. Color gradients represent the strength of the correlation coefficient. *p < .05, **p < .01, ***p < .001. https://doi.org/10.1371/journal.pone.0261151.g006 Correlations between mean ratings of feeling moved and trait empathy scores. In order to explore potential individual differences in feeling moved or touched, we calculated Pearson correlations between mean moved or touched ratings (averaged across individual time-series) and the two subscales of the Interpersonal Reactivity Index: empathic concern and fantasy. We calculated an estimate across songs by running a random effects meta-analysis for each subscale. Empathic concern correlated significantly with feeling moved or touched overall (r = .31 [.22, .41]), as well as by both sadly moving (r = .26 [.12, .41]) and joyfully moving excerpts (r = .35 [.21, .48]). Fantasy showed a smaller overall effect (r = .11 [.01, .21]), as it did not correlate significantly with feeling moved or touched by sadly moving excerpts (r = -.01 [-.16, .15]), and only modestly with feeling moved or touched by joyfully moving excerpts (r = .20 [.07, .34]). We also explored the correlations between peak feeling moved (instead of mean ratings) and trait empathy scores (see 2.3 Section in S1 File). Findings were nearly identical. Finally, since the mean ratings of feeling moved or touched revealed significant correlations with trait empathy, we decided to explore individual differences in continuous rating patterns in more detail. It may be that the higher mean values of feeling moved exhibited by the high-empathy participants either reflect higher overall levels of feeling moved, or more pronounced peaks or variations in the continuous ratings. In order to identify potential groups of people with similar rating trajectories, we performed a clustering analysis. The detailed results can be found in the 2.4 Section in S1 File. Overall, we found evidence that higher trait empathic concern was associated with higher peaks of continuous feeling moved ratings, though only for two musical excerpts: Hoppipolla and Band of Brothers. Data preparation An overview of the general data preparation process is provided in Fig 1. First, for each participants’ timestamped ratings we created time series at 1Hz resolution (one rating per second; Fig 1A). For example, if a participant started with a rating of 1 on a specific scale and changed their rating to a 3 at 30 s, the time series would show a rating of 1 from 0 s to 29 s, and a rating of 3 from 30 s until the second the rating was changed again (or until the end of the musical excerpt if the rating did not change). Second, we averaged the time series for each scale-excerpt combination across participants, resulting in seven averaged time series ratings for each of the seven excerpts (Fig 1B). Consistency of time series was computed with intraclass correlations, which represent a measure of inter-rater agreement (ICC; [62]). The mean ICC across all excerpt-rating combinations was .86 and all indices except the one for perceived sadness and Hoppipolla were above .60 (see S4 Table in S1 File), indicating good agreement among raters. Third, we decreased the resolution of the time scales by aggregating judgments within units of three consecutive seconds. We assumed that participants were not able to report their momentary experiences within the exact same second they appeared. Previous research has suggested that 3 seconds represent the temporal building blocks of many aspects related to human perceptual or motor abilities [63]. Therefore, we registered three seconds as an adequate time bin (Fig 1C). To test the robustness of our aggregation in the light of findings suggesting that aesthetic ratings are stable at 1s responses [64], we repeated the main cross-correlations for the 1Hz data. Effects were similar, though a bit stronger. An overview is provided in the (S8 Table in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Overview of the data handling process. (A) We recorded individual time series for each scale and musical excerpt. (B) Individual time series were averaged across each scale and musical excerpt combination (error bars represent 95% CIs). (C) The time series was averaged within units of three consecutive seconds. (D) We applied a cubic spline interpolation using six sampling points in order to remove trends in the time series. The depicted example graphs represent the feeling moved ratings for Explosions as an illustration of the overall process. https://doi.org/10.1371/journal.pone.0261151.g001 Fourth, we detrended the time series. Most time series are non-stationary, meaning that they show linear or higher-order increases or decreases. Comparing two non-stationary time series would inflate correlations, as they tend to cross-correlate simply due to the fact that they have trends in common. Detrending, removing the trend in time series, is one possible way to achieve the stationarity of time series data [65, 66]. Inspecting our time series, all averaged curves showed increases in ratings over time and some also decreases in ratings towards the end. More specifically, they indicated positive linear and negative quadratic trends. We detrended all time series using a cubic spline interpolation [59, 65]. Splines can be considered as an extension of polynomial regression that divides the time series into a number of k intervals, delimited by knots. Employing a cubic spline, a regression with three parameters (linear, quadratic, and cubic) is fit at each knot. We used five knots and therefore six intervals for all time series ([67]; Fig 1D). We employed two additional detrending techniques: the residual and difference methods. For the residual method we removed the linear and quadratic trends from all curves by regressing them separately in a multiple regression on an index of time in seconds and its square, and saved the unstandardized residuals (Shumway and Stoffer [66]; S5 Table in S1 File). In differencing, the difference between the current observation and the previous observation is calculated for the whole time series. The direction of all main effects was similar with the three detrending methods. However, while the differencing and spline methods produced rather similar effects, the residual method showed considerably stronger effects for most measures (see S6 Table in S1 File). Note, that we originally registered to employ the residual detrending method. However, cubic splines have been considered to be superior when it comes to removing slow drifts and have been recommended to handle time series data when collecting emotional responses [59, 68]. Results of the two additional detrending methods are reported in the S1 File. We also report the results on the original un-detrended data. While these effects showed similar direction as the detrended effects, they were much stronger, which seems to be based on the underlying trends inflating and overestimating cross-correlations. The phenomenon of feeling moved tends to happen over shorter-time intervals and is not expected to demonstrate long-term trajectories, as typically assumed for emotional experiences (in contrast to longer-lasting moods; Beedie et al. [69]). Hence for the purpose of this study detrending does not affect this significantly. Especially, the cubic spline detrending that we employ ensures that we capture these shorter-term fluctuations while removing the long-term trajectories which are more an artefact resulting from the non-stationary nature of the emotion ratings (e.g., [70]). After inspecting the time-series, we noticed that most time-series increased sharply for the first few seconds of the excerpts. Such early changes likely reflect the change from silence to music (rather than changes within the music) and have been called initial orientation time, the time it takes for continuous ratings to fall within some specific range [71]. In a previous study the median initial orientation time was observed at 8s [71]. As such sharp increases that might not reflect changes related to the music could inflate relationships across the time-series, we excluded the first three time-bins (i.e., the first nine seconds) of every excerpt. Note that this decision was not pre-registered. Acoustic feature extraction Eleven acoustic features were chosen to broadly capture the loudness-related, timbral, tonal and rhythmic aspects of the musical excerpts (see Table 1 for a description of each acoustic feature). The features can typically be classified into two categories based on the duration of the analysis-window used during the feature extraction process [72, 73]. Short-term features were extracted using a 50ms window with a 50% overlap and encompassed loudness and timbral properties. While loudness was captured by the feature root-mean-square energy, timbral features primarily comprised spectral shape descriptors, namely spectral centroid, spectral spread, spectral roll-off, entropy, roughness, and flatness. In addition, spectrotemporal fluctuations were captured by spectral flux and temporal fluctuations by zero-crossing rate. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Overview and description of acoustic features extracted. https://doi.org/10.1371/journal.pone.0261151.t001 Long-term features capture context-dependent tonal and rhythmic aspects of music and are extracted using a three second window with an overlap of 67%. Tonal variation was captured by key strength or key clarity while rhythmic variation was captured by pulse clarity. All features were extracted from each excerpt using the MIRtoolbox [78]. Subsequently, spline detrending was performed on the acoustic features, and the first 9 seconds of the excerpt were excluded from analyses as done for the emotional ratings. Finally, we created time bins of three seconds in order to match the self-report ratings. Detrending of the acoustic features was necessary to ensure comparability with the detrended self-report time series, which removed potential slow drift or long-term features. The transformation procedure was based on previous studies focusing on comparing acoustic features with neurophysiological responses [72]. Cross-correlations In order to compare changes in the different ratings over time, we computed cross-correlation functions (CCF) and their 95% confidence intervals. To calculate cross-correlations and estimate confidence intervals across sadly and joyfully moving excerpts, cross-correlations for each excerpt were employed in a random-effects meta-analysis using restricted maximum likelihood estimation in the metafor package in R [79]. Note that this approach was not pre-registered. Alternatively, we employed Fisher-Z transformations in order to obtain overall coefficients as specified in the pre-registration. Cross-correlations were first transformed using a Fisher-Z transformation, averaged, and finally back-transformed. Findings differed minimally and are presented in the (S7 Table in S1 File). We focused on cross-correlations at lag zero: A high cross-correlation between two scales at this lag means that both ratings change concurrently in the same direction as the music unfolds. Note, that a cross-correlation function at lag zero is the same as a zero-order Pearson correlation coefficient. For robustness, we also calculated the main analyses using Spearman correlation coefficients that are presented in the S9 Fig in S1 File. Similar to a previous study [53], we did not observe any systematic changes across other lags. An overview of the correlations between feeling moved or touched and all other continuous ratings, averaged for the different types of excerpts (after detrending employing the spline method) is presented in Fig 2. Additional results before detrending and after detrending employing the residual and difference methods are presented in S6 Table in S1 File. We also provide cross-correlations among the other variables in S9 Table in S1 File. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Cross-correlation functions (CCF) between feeling moved and touched and the six main variables across the musical excerpts. CCFs were calculated at lag 0 (meaning that both time series are compared at the same time) using cubic spline detrended timeseries. Sadly and joyfully moving songs are grouped separately. Overall estimates and confidence intervals are constructed employing a random-effects meta-analysis using restricted maximum likelihood estimation. Error bars represent 95% confidence intervals. RE = random effects. As heterogeneity measures we included Cochran’s Q and I2. https://doi.org/10.1371/journal.pone.0261151.g002 Warmth, chills, and a sense of connection. As predicted in H1, feeling moved or touched cross-correlated highly with experiencing warmth in the chest with an average of a cross-correlation function at lag 0 (CCF0) of .56 [.32, .80] across all excerpts. The effect was somewhat smaller for sadly than for joyfully moving excerpts (sadly: CCF0 = .50 [-.13, 1.13], joyfully: CCF0 = .61 [.47, .75]), which was based on one sadly moving excerpt showing a negative effect (Oblivion: CCF0 = -.17 [-.47, .13]). Similarly, feeling moved or touched cross-correlated highly with experiencing chills or goosebumps with an average of CCF0 = .63 [.51, .74] across all excerpts. The effect did only differ slightly between sadly (CCF0 = .65 [.45, .84]) and joyfully moving excerpts (CCF0 = .60 [.45, .76]). Finally, feeling moved or touched and feeling a sense of connectedness correlated highly with an average of CCF0 = .66 [.54, .76] across all excerpts. The effect was significantly stronger for joyfully (CCF0 = .75 [.63, .86]) than sadly moving excerpts (CCF0 = .51 [.39, .64]). Perceived beauty. As predicted in H2, feeling moved or touched cross-correlated highly with perceived beauty with an average of CCF0 = .60 [.49, .71] across all excerpts. The sadly moving excerpts (CCF0 = .58 [.32, .83]) showed similar effects compared to the joyfully moving excerpts (CCF0 = .60 [.47, .74]), though higher variation across excerpts. The averaged (non-detrended) continuous ratings of feeling moved or touched, perceived beauty, and feeling a sense of connection for the sadly and joyfully moving excerpts are displayed in Fig 3 and the detrended (cubic spline) ratings are presented in the S6 Fig in S1 File. The (non-detrended) ratings of feeling moved or touched, chills, and warmth in the chest are presented in Fig 4 (and the detrended ratings in S7 Fig in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. The averaged non-detrended continuous ratings of feeling moved or touched, perceived beauty, and feeling a sense of connection for the sadly (Upper Row) and joyfully moving (Lower Row) music excerpts. Ratings were provided on a 5-point scale (ranging from 1 (not at all) to 5 (extremely)) and aggregated in 3s time bins. https://doi.org/10.1371/journal.pone.0261151.g003 Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. The averaged (non-detrended) continuous ratings of feeling moved or touched, warmth in the chest, and chills for the sadly (Upper Row) and joyfully moving (Lower Row) music excerpts. Ratings were provided on a 5-point scale (ranging from 1 (not at all) to 5 (extremely)) and aggregated in 3s time bins. https://doi.org/10.1371/journal.pone.0261151.g004 Perceived sadness. The cross-correlation between feeling moved or touched and perceiving the excerpt as sad was CCF0 = .02 [-.34, .38] across all musical excerpts. As hypothesized in H3, the effect was much stronger for the sadly moving excerpts (CCF0 = .49 [.23, .76]) than for the joyfully moving excerpts (CCF0 = -.33 [-.59, -.07]), which showed an effect in the opposite direction. Perceived joy. The cross-correlation between feeling moved or touched and perceiving the excerpt as joyful was CCF0 = .61 [.48, .75] across all excerpts. Partly confirming predictions in H4, the effect was somewhat stronger for joyfully moving (CCF0 = .67 [.59, .74]) than sadly moving excerpts (CCF0 = .49 [.08, .90]), although this difference was driven by greater variation across the sadly moving excerpts. Except for one sadly moving excerpt (Oblivion: CCF0 = .05 [-.26, .35]), all musical excerpts showed a strong positive cross-correlation above .59. The averaged (non-detrended) continuous ratings of feeling moved or touched, perceived sadness, and perceived joy for the sadly and joyfully moving excerpts are displayed in Fig 5 and the detrended (cubic spline) ratings are presented in the (S8 Fig in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. The averaged (non-detrended) continuous ratings of feeling moved or touched, perceived sadness, and perceived joy for the sadly (Upper Row) and joyfully moving (Lower Row) music excerpts. Ratings were provided on a 5-point scale (ranging from 1 (not at all) to 5 (extremely)) and aggregated in 3s time bins. https://doi.org/10.1371/journal.pone.0261151.g005 Warmth, chills, and a sense of connection. As predicted in H1, feeling moved or touched cross-correlated highly with experiencing warmth in the chest with an average of a cross-correlation function at lag 0 (CCF0) of .56 [.32, .80] across all excerpts. The effect was somewhat smaller for sadly than for joyfully moving excerpts (sadly: CCF0 = .50 [-.13, 1.13], joyfully: CCF0 = .61 [.47, .75]), which was based on one sadly moving excerpt showing a negative effect (Oblivion: CCF0 = -.17 [-.47, .13]). Similarly, feeling moved or touched cross-correlated highly with experiencing chills or goosebumps with an average of CCF0 = .63 [.51, .74] across all excerpts. The effect did only differ slightly between sadly (CCF0 = .65 [.45, .84]) and joyfully moving excerpts (CCF0 = .60 [.45, .76]). Finally, feeling moved or touched and feeling a sense of connectedness correlated highly with an average of CCF0 = .66 [.54, .76] across all excerpts. The effect was significantly stronger for joyfully (CCF0 = .75 [.63, .86]) than sadly moving excerpts (CCF0 = .51 [.39, .64]). Perceived beauty. As predicted in H2, feeling moved or touched cross-correlated highly with perceived beauty with an average of CCF0 = .60 [.49, .71] across all excerpts. The sadly moving excerpts (CCF0 = .58 [.32, .83]) showed similar effects compared to the joyfully moving excerpts (CCF0 = .60 [.47, .74]), though higher variation across excerpts. The averaged (non-detrended) continuous ratings of feeling moved or touched, perceived beauty, and feeling a sense of connection for the sadly and joyfully moving excerpts are displayed in Fig 3 and the detrended (cubic spline) ratings are presented in the S6 Fig in S1 File. The (non-detrended) ratings of feeling moved or touched, chills, and warmth in the chest are presented in Fig 4 (and the detrended ratings in S7 Fig in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. The averaged non-detrended continuous ratings of feeling moved or touched, perceived beauty, and feeling a sense of connection for the sadly (Upper Row) and joyfully moving (Lower Row) music excerpts. Ratings were provided on a 5-point scale (ranging from 1 (not at all) to 5 (extremely)) and aggregated in 3s time bins. https://doi.org/10.1371/journal.pone.0261151.g003 Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. The averaged (non-detrended) continuous ratings of feeling moved or touched, warmth in the chest, and chills for the sadly (Upper Row) and joyfully moving (Lower Row) music excerpts. Ratings were provided on a 5-point scale (ranging from 1 (not at all) to 5 (extremely)) and aggregated in 3s time bins. https://doi.org/10.1371/journal.pone.0261151.g004 Perceived sadness. The cross-correlation between feeling moved or touched and perceiving the excerpt as sad was CCF0 = .02 [-.34, .38] across all musical excerpts. As hypothesized in H3, the effect was much stronger for the sadly moving excerpts (CCF0 = .49 [.23, .76]) than for the joyfully moving excerpts (CCF0 = -.33 [-.59, -.07]), which showed an effect in the opposite direction. Perceived joy. The cross-correlation between feeling moved or touched and perceiving the excerpt as joyful was CCF0 = .61 [.48, .75] across all excerpts. Partly confirming predictions in H4, the effect was somewhat stronger for joyfully moving (CCF0 = .67 [.59, .74]) than sadly moving excerpts (CCF0 = .49 [.08, .90]), although this difference was driven by greater variation across the sadly moving excerpts. Except for one sadly moving excerpt (Oblivion: CCF0 = .05 [-.26, .35]), all musical excerpts showed a strong positive cross-correlation above .59. The averaged (non-detrended) continuous ratings of feeling moved or touched, perceived sadness, and perceived joy for the sadly and joyfully moving excerpts are displayed in Fig 5 and the detrended (cubic spline) ratings are presented in the (S8 Fig in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. The averaged (non-detrended) continuous ratings of feeling moved or touched, perceived sadness, and perceived joy for the sadly (Upper Row) and joyfully moving (Lower Row) music excerpts. Ratings were provided on a 5-point scale (ranging from 1 (not at all) to 5 (extremely)) and aggregated in 3s time bins. https://doi.org/10.1371/journal.pone.0261151.g005 Exploratory analyses The analyses reported in the following section were not pre-registered and/or did not test a pre-registered hypothesis, and should thus be considered exploratory. Enjoyment, familiarity and self-reported tears. In order to explore the relationship between liking, familiarity, and experiencing tears or moist eyes, liking of the musical excerpt was regressed on familiarity, tears or moist eyes and their interaction in a multilevel model. Intercepts were allowed to vary randomly according to participant and excerpt type. First, we found a main effect of familiarity. Higher familiarity with an excerpt increased the liking of that excerpt with an unstandardized coefficient of B = -.42 [-.53, -.32], t(2380) = -8.05, p < .001. In addition, we observed an interaction effect of familiarity and tears, B = .08 [.03, .12], t(2292) = 3.33, p < .001. Highly familiar excerpts were generally liked more. However, when combined with a strong response of tears, low familiarity resulted in more enjoyment than high familiarity and no or medium tear response. Acoustic and musical correlates of feeling moved. Spearman correlations were calculated between the detrended continuous ratings of feeling moved or touched and the eleven acoustic features (extracted using the MIRtoolbox). Fisher’s [80] Z transformation was then applied to the correlation values followed by adjusting by the factor 1 / √(df − 3), where df represents the estimated number of effective degrees of freedom calculated using the approach described by Pyper and Peterman [81]. The subsequent corrected z value was then converted to a respective p value. This procedure is the same as that carried out in previous studies focusing on the correlation between acoustic features and continuous neural responses [72]. The results of the correlation analyses are displayed in Fig 6. Joyfully moving excerpts exhibited similar positive correlation profiles between feeling moved or touched and the acoustic features representing loudness (rms), sensory dissonance (roughness) and spectrotemporal variations (flux). In addition, Vltava displayed a significant positive correlation with zero-crossing rate (zcr) and a negative correlation with key clarity, while Hoppipolla displayed a positive correlation with spectral rolloff. However, among the three sadly moving excerpts, only Allegri displayed significant correlations of feeling moved or touched with zero-crossing rate (zcr) and spectral entropy, with Oblivion displaying a similar correlation profile. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 6. A visualization of the spearman correlations between continuous ratings of feeling moved or touched, and the acoustic and musical features of the music excerpts. Color gradients represent the strength of the correlation coefficient. *p < .05, **p < .01, ***p < .001. https://doi.org/10.1371/journal.pone.0261151.g006 Correlations between mean ratings of feeling moved and trait empathy scores. In order to explore potential individual differences in feeling moved or touched, we calculated Pearson correlations between mean moved or touched ratings (averaged across individual time-series) and the two subscales of the Interpersonal Reactivity Index: empathic concern and fantasy. We calculated an estimate across songs by running a random effects meta-analysis for each subscale. Empathic concern correlated significantly with feeling moved or touched overall (r = .31 [.22, .41]), as well as by both sadly moving (r = .26 [.12, .41]) and joyfully moving excerpts (r = .35 [.21, .48]). Fantasy showed a smaller overall effect (r = .11 [.01, .21]), as it did not correlate significantly with feeling moved or touched by sadly moving excerpts (r = -.01 [-.16, .15]), and only modestly with feeling moved or touched by joyfully moving excerpts (r = .20 [.07, .34]). We also explored the correlations between peak feeling moved (instead of mean ratings) and trait empathy scores (see 2.3 Section in S1 File). Findings were nearly identical. Finally, since the mean ratings of feeling moved or touched revealed significant correlations with trait empathy, we decided to explore individual differences in continuous rating patterns in more detail. It may be that the higher mean values of feeling moved exhibited by the high-empathy participants either reflect higher overall levels of feeling moved, or more pronounced peaks or variations in the continuous ratings. In order to identify potential groups of people with similar rating trajectories, we performed a clustering analysis. The detailed results can be found in the 2.4 Section in S1 File. Overall, we found evidence that higher trait empathic concern was associated with higher peaks of continuous feeling moved ratings, though only for two musical excerpts: Hoppipolla and Band of Brothers. Enjoyment, familiarity and self-reported tears. In order to explore the relationship between liking, familiarity, and experiencing tears or moist eyes, liking of the musical excerpt was regressed on familiarity, tears or moist eyes and their interaction in a multilevel model. Intercepts were allowed to vary randomly according to participant and excerpt type. First, we found a main effect of familiarity. Higher familiarity with an excerpt increased the liking of that excerpt with an unstandardized coefficient of B = -.42 [-.53, -.32], t(2380) = -8.05, p < .001. In addition, we observed an interaction effect of familiarity and tears, B = .08 [.03, .12], t(2292) = 3.33, p < .001. Highly familiar excerpts were generally liked more. However, when combined with a strong response of tears, low familiarity resulted in more enjoyment than high familiarity and no or medium tear response. Acoustic and musical correlates of feeling moved. Spearman correlations were calculated between the detrended continuous ratings of feeling moved or touched and the eleven acoustic features (extracted using the MIRtoolbox). Fisher’s [80] Z transformation was then applied to the correlation values followed by adjusting by the factor 1 / √(df − 3), where df represents the estimated number of effective degrees of freedom calculated using the approach described by Pyper and Peterman [81]. The subsequent corrected z value was then converted to a respective p value. This procedure is the same as that carried out in previous studies focusing on the correlation between acoustic features and continuous neural responses [72]. The results of the correlation analyses are displayed in Fig 6. Joyfully moving excerpts exhibited similar positive correlation profiles between feeling moved or touched and the acoustic features representing loudness (rms), sensory dissonance (roughness) and spectrotemporal variations (flux). In addition, Vltava displayed a significant positive correlation with zero-crossing rate (zcr) and a negative correlation with key clarity, while Hoppipolla displayed a positive correlation with spectral rolloff. However, among the three sadly moving excerpts, only Allegri displayed significant correlations of feeling moved or touched with zero-crossing rate (zcr) and spectral entropy, with Oblivion displaying a similar correlation profile. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 6. A visualization of the spearman correlations between continuous ratings of feeling moved or touched, and the acoustic and musical features of the music excerpts. Color gradients represent the strength of the correlation coefficient. *p < .05, **p < .01, ***p < .001. https://doi.org/10.1371/journal.pone.0261151.g006 Correlations between mean ratings of feeling moved and trait empathy scores. In order to explore potential individual differences in feeling moved or touched, we calculated Pearson correlations between mean moved or touched ratings (averaged across individual time-series) and the two subscales of the Interpersonal Reactivity Index: empathic concern and fantasy. We calculated an estimate across songs by running a random effects meta-analysis for each subscale. Empathic concern correlated significantly with feeling moved or touched overall (r = .31 [.22, .41]), as well as by both sadly moving (r = .26 [.12, .41]) and joyfully moving excerpts (r = .35 [.21, .48]). Fantasy showed a smaller overall effect (r = .11 [.01, .21]), as it did not correlate significantly with feeling moved or touched by sadly moving excerpts (r = -.01 [-.16, .15]), and only modestly with feeling moved or touched by joyfully moving excerpts (r = .20 [.07, .34]). We also explored the correlations between peak feeling moved (instead of mean ratings) and trait empathy scores (see 2.3 Section in S1 File). Findings were nearly identical. Finally, since the mean ratings of feeling moved or touched revealed significant correlations with trait empathy, we decided to explore individual differences in continuous rating patterns in more detail. It may be that the higher mean values of feeling moved exhibited by the high-empathy participants either reflect higher overall levels of feeling moved, or more pronounced peaks or variations in the continuous ratings. In order to identify potential groups of people with similar rating trajectories, we performed a clustering analysis. The detailed results can be found in the 2.4 Section in S1 File. Overall, we found evidence that higher trait empathic concern was associated with higher peaks of continuous feeling moved ratings, though only for two musical excerpts: Hoppipolla and Band of Brothers. Discussion The purpose of this study was to explore continuous ratings of feeling moved or touched by music, and how the time course of feeling moved might covary with specific appraisals, emotional perceptions, and bodily sensations. In addition, we explored the association of continuous ratings of feeling moved with acoustical features extracted from the musical excerpts. Building on previous theorizing and studies on feeling moved (e.g., [6, 22]), we hypothesized that ratings of feeling moved would cross-correlate with feeling a sense of connection, experiencing chills, and experiencing a warm feeling in the chest (H1). Furthermore, because previous studies associated feeling moved with the perceived beauty of music [7], we hypothesized that feeling moved or touched would cross-correlate with ratings of perceived beauty (H2). Finally, we hypothesized that feeling moved or touched would cross-correlate with perceived sadness in the case of sadly moving excerpts (and less so for happily moving excerpts; H3), while it would cross-correlate with perceived happiness for joyfully moving excerpts (and less so for sadly moving excerpts; H4). Most of our pre-registered hypotheses were supported by our findings. Figs 4–6 show that the time course of feeling moved largely coincided with that of sense of connection, chills, warmth in the chest, perceived beauty and perceived joy across all musical excerpts, and with perceived sadness for sadly moving excerpts. The preregistered and additional cross-correlation analyses confirmed these impressions. A further inspection of the time courses of these ratings shows that they typically increase along the course of the excerpt, with a rather strong initial rise within the first 30 seconds, resulting in a positive linear trends in all ratings and excerpts. Notably, we cannot completely disentangle whether such a rise was caused by actual sudden changes in experiences or by a general need to change the rating because it always started at the lowest score. Furthermore, some excerpts show only monotonic increases, while others had peak ratings before the end and then declining ratings, and some more than one peak. These different time courses are not necessarily reflective of the dynamics of whole musical pieces, as we only used specific excerpts. They are similar in shape to the ones found for moving videos [53]. Continuous ratings of feeling moved or touched had one of the highest cross-correlations overall with ratings of “feeling a sense of connection”, suggesting that appraisals or experiences of closeness or affiliation are associated with experiences of feeling moved also in the case of music. In contrast to a previous study in which people rated the perceived closeness of the people appearing in the video clips [53], we chose to operationalize closeness more broadly as “feeling a sense of connection”. The exact nature of the connection experienced was intentionally left open in order to accommodate for the “floating intentionality” inherent in music [82]. Thus, it is possible that this ‘sense of connection’ may have been construed as existing between the listener and another agent (the music, the performer, other listeners, or perhaps humanity in general), or between real or virtual agents perceived in the music (cf. [8]). In terms of the intensification of communal sharing relations central to kama muta theory, these ‘connections’ could be conceptualized in terms of first- or second-person empathy and compassion (e.g., empathizing with the music or experiencing the music as empathizing with oneself; see [9, 83, 84]), or in terms of affiliative intentions perceived in the music [8]. Despite the range of possible relationships and interpretations, our results demonstrate that participants rated “feeling a sense of connection” consistently, suggesting a shared understanding of the concept and the cues contributing to it. Our findings also confirmed that ratings of feeling moved or touched cross-correlated with self-reported chills or goosebumps and experiencing a warm feeling in the chest in response to music, replicating previous findings on the association of these variables (e.g., [5, 22, 43, 45]). These patterns of cross-correlations were similar across both sadly and joyfully moving excerpts. This indicates that feeling moved by music is associated with a similar pattern of physiological or bodily sensations as feeling moved by videos depicting social scenarios (and experiences of feeling moved in general; [5]), and that this pattern does not depend on the perceived sadness or joyfulness of the music. Interestingly, while Koneçni [18, 19] predicts chills to be a more frequent response to music than feeling moved, we found high cross-correlations between the two, suggesting that they largely coincided in occurrence. Furthermore, in line with a previous study showing that movingness mediates the relationship between perceived sadness and beauty in music [7], we found positive cross-correlations between ratings of feeling moved or touched and perceived beauty. Interestingly, these cross-correlations were consistent across both sadly and joyfully moving excerpts. Previous research has associated perceived beauty with perceived sadness and only to a lower degree with perceived happiness (r = .59 vs. r = .16; Eerola and Vuoskoski [85]), while in the current study we found overall weaker associations between perceived sadness and beauty than between perceived joy and beauty (r = .12 vs. r = .54). Supporting previous findings [86], moving and touching music was also perceived as beautiful. This bears the question whether feeling moved by music triggers a perception of beauty or vice versa. Is all music experienced as beautiful automatically experienced as moving or is moving music automatically perceived as beautiful? Further research is needed to investigate this possibility. Finally, we had hypothesized that feeling moved or touched would cross-correlate with perceived joy for joyfully moving excerpts but less so for sadly moving excerpts, and with perceived sadness for sadly moving excerpts but less so for joyfully moving excerpts. While feeling moved cross-correlated with perceived joy, but not sadness for joyfully moving music, the picture regarding sadly moving music turned out to be somewhat more complex: When the overall emotional tone of the music was sad rather than joyful, feeling moved or touched cross-correlated with both perceived sadness and perceived joy (except for the excerpt “Oblivion”). The finding that ratings of feeling moved or touched cross-correlated positively with perceived joy across both joyfully and sadly moving excerpts, supports previous theories suggesting that the overall tonality of the experience of feeling moved is positive [6]. This is in fact reflected in most perspectives on feeling moved (cf. [87]). The finding that feeling moved or touched cross-correlated positively with perceived sadness for sadly moving excerpts can be interpreted from two different angles. First, it could mean that, although predominantly positive, feeling moved constitutes a mixed state including both negative and positive affect, as suggested by one specific theory [21]. Second, this could mean that feeling moved is experienced as predominantly positive but can co-occur with other negative emotions such as sadness, as suggested by kama muta theory [12, 15]. The observation that feeling moved or touched cross-correlated negatively with perceived sadness for joyfully moving excerpts provides some evidence for the latter explanation. However, it should be noted that we assessed perceived and not experienced sadness and joy, which means that our findings can only be interpreted as indirect evidence. In sum, the main findings support the idea that feeling moved is perceived as predominantly positive—also in the context of music. Not only did we obtain strong cross-correlations between feeling moved or touched and the other variables (except for perceived sadness in the joyfully moving and overall context), we also observed strong associations among perceived joy, a sense of connection, and reported chills and warmth. These findings support kama muta theory [15] and the theory by Menninghaus et al. [21]. Kama muta theory argues that the underlying emotional construct represents the co-occurrence of sudden intensifications of communal sharing relationships, experiences of tears, chills, and/or warmth in the chest, motivations to act on one’s communal sharing relations, positive valence, and labels such as moved or touched [15]. While the kama muta framework has been supported for short film clips [5], this is the first systematic evidence for the configuration of kama muta in response to music (see [31]). Notably, we did not assess the motivational component of kama muta as it is currently unclear how this might relate to the reception of music. A further theory conceptualizing experiences of feeling moved is core values theory [88]. Core values are defined as those that are of central importance to a social group. Connection is presumably of central importance to most groups, but so are other values such as achievement [89]. Our findings are thus compatible with the core values theory of feeling moved but do not provide evidence that values other than connection contribute to being moved by music. Our paradigm could be used to assess perceived skill or virtuosity of the musicians and study its cross-correlation with feeling moved. According to core values theory, there should be a positive cross-correlation to the extent that being a skillful musician is seen as an admirable value in that cultural context. Finally, aesthetic trinity theory [18, 19] conceptualizes feeling moved as a highly idiosyncratic response to the sublime, and chills as a more frequent response to music. Our results lend partial support to that theory, as ‘sense of connection’ can refer to connection with the sublime. However, given previous work on music and the sense of connection it conveys (e.g., [90]), we suggest that it mainly refers to social connection perceived between the self and the music or between agents perceived in the music. Furthermore, we found high inter-rater agreements for all scales. The ICC for sense of connection was .84 or higher for all musical excerpts studied, suggesting that there was high interpersonal agreement on which passages conveyed more connection than others. This suggests that the experience of feeling moved by music can be reliably evoked by the same musical passages across participants. It is worth noting that we had chosen these excerpts to be moving. Other musical pieces can be expected to result in lower inter-rater agreements, and thus more idiosyncratic experiences of feeling moved. Still, our findings suggest that feeling moved can be reliably evoked by particular musical features, independent of individual recollections and associations. Acoustic correlates of feeling moved Our exploratory analysis of the acoustic and musical correlates of continuous ratings of feeling moved or touched revealed a pattern of acoustic correlates for the joyfully moving excerpts, while no consistent patterns were observed for the sadly moving excerpts. Continuous ratings of feeling moved or touched correlated positively and significantly with loudness (rms energy) in the case of all joyfully moving excerpts. For three of the four joyful excerpts, feeling moved or touched also correlated with spectral roughness (or sensory dissonance). In the case of the excerpts used in the present study, higher values in this feature likely correspond to increased complexity of the spectral content, reflecting multiple instruments sounding together. For two of the joyful excerpts, Hoppipolla and Vltava, feeling moved or touched also correlated positively with spectral flux. Previous work investigating the acoustic correlates of felt and perceived emotions in music has associated loudness, spectral flux and roughness with the arousal dimension of affect (e.g., [91, 92]). It may be that, at least in the case of joyfully moving music, arousal-related musical features contribute to feeling moved by increasing felt arousal. However, in the case of the sadly moving excerpts, feeling moved or touched only correlated significantly with zero-crossing rate and spectral entropy, and only in the case of one excerpt—Allegri. Zero-crossing rate generally reflects rapid fluctuations in the temporal domain and has been broadly used as a feature to detect the presence of vocals or voice-like sounds and voice activity [93, 94]. Specifically, sustained sounds tend to have greater zero-crossing rate than percussive sounds. Allegri is characterized mainly by sustained vocals, while Vltava and Oblivion are dominated by string-instruments that carry the melody in a continuous fashion. Overall, it is possible that arousal is less important for feeling moved evoked by sad (compared to joyful) music, but future studies should specifically investigate whether feeling moved by sad versus joyful music is associated with different levels of psychophysiological arousal. On the other hand, timbre or choice of instruments might be more important for feeling moved by sad music [95]. Moreover, several studies have emphasized the importance of lyrics especially in sad music contributing to feeling moved [96]. This warrants further study. Importantly, the current findings are based on a small number of musical excerpts so caution should be applied when interpreting these findings, especially regarding the inconsistent results for sadly moving excerpts. In addition, since continuous ratings of feeling moved cross-correlated strongly with perceived joy and perceived beauty, it is possible that the observed acoustic features are not specific to feeling moved, but rather reflect co-occurrences with other evaluations or experiences. Altogether, we observed correlational evidence that can guide future studies in experimentally investigating different musical aspects and measuring ratings of feeling moved (e.g., [39]). Trait empathy and feeling moved by music Finally, we explored individual differences in feeling moved or touched by music. Empathic concern, the tendency to respond sympathetically to others in need [57], correlated positively with mean feeling moved or touched ratings in the case of both joyfully and sadly moving excerpts. This finding corroborates and extends previous work that has associated empathic concern with feeling moved or kama muta in response to videos, sad music, and other stimuli [7, 52, 58]. In an auxiliary analysis, we also found that empathic concern was associated with higher peaks of feeling moved—though only for two joyfully moving musical pieces, which should be interpreted with caution. Recent studies found that empathic concern was the only subscale of trait empathy (as assessed by the IRI; Davis [57]) that was consistently associated with feeling moved (as well as reported tears, chills, and warmth) in response to videos and written narratives [5, 58]. The present findings suggest that a similar relationship exists for the case of music. Zickfeld et al. [58] argued that state empathic concern could be considered as a special case of intensifications of communal sharing and thereby a type of feeling moved or kama muta. Such a relationship could explain the cross-correlation of feeling moved with perceived sadness. Perceiving someone in need can be evaluated and experienced as negative or also sad, whereas sympathizing with the needy person can be considered as intensifying one’s communal bonds, thereby inducing feeling moved [58]. How this relates to the context of music is less clear, although recent work suggests that listeners may experience feelings of compassion when listening to sad music [84]. The current results suggest an important role of empathic processes in emotional responses to moving music (see [83, 84]), and are in line with the view of musical experience as inherently social (e.g., [36]). Previous studies have argued that empathic engagement with music could take the form of resonating with the expressions and imagined experiences of the performer or the composer (e.g., [27]), or identifying with an imagined narrative or a virtual persona represented by the music [97]. Fantasy, the ability to transpose oneself into fictional situations or stories, another facet of trait empathy that we assessed [57], has previously been associated with feeling moved by sad music [7, 52]. However, in the present study, fantasy only showed stronger correlations with ratings of feeling moved or touched in response to joyful music, but less so in response to sad music. One difference between the previous studies versus the current study is that we averaged all feeling moved ratings across the whole time series, whereas previous studies used a summary rating of feeling moved after listening to the piece. Thus, it may be that individuals with higher levels of fantasy are not more moved by sad music throughout, but rather respond more strongly to highly moving passages, which may be better reflected in their own summary judgement rather than in averaging across the whole piece. Furthermore, they may remember these peak moments better than people lower in fantasy, or reconstruct the piece as more moving when asked about it in retrospect, thus providing a higher rating. By systematically comparing continuous with summary ratings, future research can directly test these propositions and quantify the contribution of traits, musical features and memory processes to ratings of feeling moved or touched. Limitations and future directions An obvious limitation of the present study is that only seven music excerpts were used as stimuli. However, this aspect of the experiment design was closely tied to the constraints of the continuous rating paradigm. This limitation is particularly relevant for the analysis of the acoustic and musical features contributing to feeling moved, since the pattern of results may be specific to the particular music excerpts. It is likely that the acoustic and musical characteristics associated with feeling moved vary somewhat from piece to piece, and thus our findings reflect only a part of the entire picture. Moreover, the musical features that were explored in the present study do not represent an exhaustive selection of possible musical features, and thus there may be other, additional musical or acoustic features that contribute to feeling moved or touched. It is also likely that the relationship between musical features and perceptual ratings is non-linear, limiting the explanatory power of linear analysis methods such as those used in the present study. Similarly, we observed higher heterogeneity in effects for the sadly moving excerpts in comparison to the happily moving excerpts. This was mostly driven by Oblivion that typically showed smaller effects than the other two sadly moving excerpts, but sometimes even the opposite as for example for the relationship between feeling moved or touched and warmth in the chest. It is unclear whether that particular excerpt represents a more prototypical or more atypical version of the category of sadly moving music. Future studies employing a larger stimulus pool could remedy this shortcoming. The generalizability of the current findings hinges not only on the limited stimulus pool, but also on the employed sample. Amazon MTurk participants have been repeatedly criticized for providing low-quality data because they do not invest sufficient effort, are non-naive, and less trustworthy (e.g., [98]). However, the evidence for these claims seems to be rather mixed and dependent on specific situations [99]. Hauser et al. [99] highlight that high-quality data can be collected from MTurk, given specific research designs and an increased focus on attention checks. In the present study, we employed several attention checks and tried to make sure that participants understood the main task in order to avoid the possible pitfalls associated with MTurk participants. The fact that we replicated the main findings from a previous study using a similar paradigm [53] also speaks to the validity of our results. Nevertheless, we acknowledge that conclusions beyond the present sample and also regarding possible cross-cultural differences in being moved responses (see [5]) are necessarily limited. However, our findings are an important first step towards understanding the contributing musical features, as well as the extent to which these may vary between sadly and joyfully moving music. Future studies could investigate the contributing features in a larger and more varied set of music examples and more diverse cross-cultural populations. A possibility for targeting such larger stimuli sets could be to instruct participants to self-select moving music (e.g., [43, 100]). While such a design minimizes the control over the stimulus material, it would be interesting to see whether the current findings replicate when employing such a technique. Furthermore, other types of analyses and experiments might shed more light on the musical cues and appraisals that are associated with feeling moved. For example, is feeling moved dependent on experiencing music as conveying prosocial intentions, or perceiving musical events in terms of prosocial interactions between agents (cf. [8])? Another limitation is related to the continuous rating paradigm itself, where the participants are asked to continuously monitor their subjective experience and adjust their ratings accordingly. It may be that the additional cognitive load of monitoring one’s internal states might detract from the intensity of the experience itself. We tried to mitigate the potential effects of this possibility by having participants only use one continuous scale per excerpt, and by using a Likert-type scale with 5 steps rather than a visual analogue slider, for example, making it less likely for participants to have to constantly adjust their rating. A complementary approach for future investigations could be to measure psychophysiological responses with and without continuous ratings, for example, in order to obtain an additional index of emotional arousal. This could also help with identifying peaks in physiological arousal, and whether these coincide with self-reported emotional peaks. Based on our design we were not able to assess intra-individual processes, as participants never rated more than one scale for the same musical piece. All presented comparisons were made on an inter-individual level. While it is intriguing to see the amount of convergence for averaged ratings of different individuals for the same musical excerpt, this approach might have overestimated correlations among the different ratings, as within error variance remains undetected. We also included two musical excerpts that featured vocals. While none of the participants were proficient in the depicted languages, the human voice is known to transport specific emotions as well as empathy (e.g., [101]). Nevertheless, the two vocal pieces showed similar results as the instrumental excerpts. In addition, we assessed perceived joy and sadness in the music, and not experienced joy and sadness induced by the music. Previous studies [7] and our pilot study suggested that ratings of perceived and experienced emotions in music correlate highly. Nonetheless, it should be emphasized that our measures only allow for indirect inferences about participants’ experiences of joy and sadness. Finally, in contrast to previous studies employing dichotomous ratings (e.g., [43]) we measured subjective chills using a continuous scale (based on Schubert et al. [53]). This was done in order to assess different intensities of chills or goosebumps. Importantly, we might therefore consider responses that would not be classified as chills responses with dichotomous rating paradigms. This is also obvious in the rather long average duration of chills ratings above the lowest scale point, suggesting that we not only covered peak-states but also lower intensities of subjective chills. Overall, we replicated associations between chills and different variables such as feeling moved, pointing to the validity of this measure. Notably, previous studies also differ in the reported average length of chills responses [43, 102], which is probably related to different methodologies and the exact operationalization of the phenomenon (i.e., as chills, goosebumps, or thrills). It is possible that for example thrills refer to a higher intensity response than goosebumps or that several short chill episodes occurring in quick succession are perceived as one longer response. A more standardized definition of the phenomenon of chills (see [44]) would be helpful in comparing different findings and evaluating the degree to which they investigate the same phenomenon. Acoustic correlates of feeling moved Our exploratory analysis of the acoustic and musical correlates of continuous ratings of feeling moved or touched revealed a pattern of acoustic correlates for the joyfully moving excerpts, while no consistent patterns were observed for the sadly moving excerpts. Continuous ratings of feeling moved or touched correlated positively and significantly with loudness (rms energy) in the case of all joyfully moving excerpts. For three of the four joyful excerpts, feeling moved or touched also correlated with spectral roughness (or sensory dissonance). In the case of the excerpts used in the present study, higher values in this feature likely correspond to increased complexity of the spectral content, reflecting multiple instruments sounding together. For two of the joyful excerpts, Hoppipolla and Vltava, feeling moved or touched also correlated positively with spectral flux. Previous work investigating the acoustic correlates of felt and perceived emotions in music has associated loudness, spectral flux and roughness with the arousal dimension of affect (e.g., [91, 92]). It may be that, at least in the case of joyfully moving music, arousal-related musical features contribute to feeling moved by increasing felt arousal. However, in the case of the sadly moving excerpts, feeling moved or touched only correlated significantly with zero-crossing rate and spectral entropy, and only in the case of one excerpt—Allegri. Zero-crossing rate generally reflects rapid fluctuations in the temporal domain and has been broadly used as a feature to detect the presence of vocals or voice-like sounds and voice activity [93, 94]. Specifically, sustained sounds tend to have greater zero-crossing rate than percussive sounds. Allegri is characterized mainly by sustained vocals, while Vltava and Oblivion are dominated by string-instruments that carry the melody in a continuous fashion. Overall, it is possible that arousal is less important for feeling moved evoked by sad (compared to joyful) music, but future studies should specifically investigate whether feeling moved by sad versus joyful music is associated with different levels of psychophysiological arousal. On the other hand, timbre or choice of instruments might be more important for feeling moved by sad music [95]. Moreover, several studies have emphasized the importance of lyrics especially in sad music contributing to feeling moved [96]. This warrants further study. Importantly, the current findings are based on a small number of musical excerpts so caution should be applied when interpreting these findings, especially regarding the inconsistent results for sadly moving excerpts. In addition, since continuous ratings of feeling moved cross-correlated strongly with perceived joy and perceived beauty, it is possible that the observed acoustic features are not specific to feeling moved, but rather reflect co-occurrences with other evaluations or experiences. Altogether, we observed correlational evidence that can guide future studies in experimentally investigating different musical aspects and measuring ratings of feeling moved (e.g., [39]). Trait empathy and feeling moved by music Finally, we explored individual differences in feeling moved or touched by music. Empathic concern, the tendency to respond sympathetically to others in need [57], correlated positively with mean feeling moved or touched ratings in the case of both joyfully and sadly moving excerpts. This finding corroborates and extends previous work that has associated empathic concern with feeling moved or kama muta in response to videos, sad music, and other stimuli [7, 52, 58]. In an auxiliary analysis, we also found that empathic concern was associated with higher peaks of feeling moved—though only for two joyfully moving musical pieces, which should be interpreted with caution. Recent studies found that empathic concern was the only subscale of trait empathy (as assessed by the IRI; Davis [57]) that was consistently associated with feeling moved (as well as reported tears, chills, and warmth) in response to videos and written narratives [5, 58]. The present findings suggest that a similar relationship exists for the case of music. Zickfeld et al. [58] argued that state empathic concern could be considered as a special case of intensifications of communal sharing and thereby a type of feeling moved or kama muta. Such a relationship could explain the cross-correlation of feeling moved with perceived sadness. Perceiving someone in need can be evaluated and experienced as negative or also sad, whereas sympathizing with the needy person can be considered as intensifying one’s communal bonds, thereby inducing feeling moved [58]. How this relates to the context of music is less clear, although recent work suggests that listeners may experience feelings of compassion when listening to sad music [84]. The current results suggest an important role of empathic processes in emotional responses to moving music (see [83, 84]), and are in line with the view of musical experience as inherently social (e.g., [36]). Previous studies have argued that empathic engagement with music could take the form of resonating with the expressions and imagined experiences of the performer or the composer (e.g., [27]), or identifying with an imagined narrative or a virtual persona represented by the music [97]. Fantasy, the ability to transpose oneself into fictional situations or stories, another facet of trait empathy that we assessed [57], has previously been associated with feeling moved by sad music [7, 52]. However, in the present study, fantasy only showed stronger correlations with ratings of feeling moved or touched in response to joyful music, but less so in response to sad music. One difference between the previous studies versus the current study is that we averaged all feeling moved ratings across the whole time series, whereas previous studies used a summary rating of feeling moved after listening to the piece. Thus, it may be that individuals with higher levels of fantasy are not more moved by sad music throughout, but rather respond more strongly to highly moving passages, which may be better reflected in their own summary judgement rather than in averaging across the whole piece. Furthermore, they may remember these peak moments better than people lower in fantasy, or reconstruct the piece as more moving when asked about it in retrospect, thus providing a higher rating. By systematically comparing continuous with summary ratings, future research can directly test these propositions and quantify the contribution of traits, musical features and memory processes to ratings of feeling moved or touched. Limitations and future directions An obvious limitation of the present study is that only seven music excerpts were used as stimuli. However, this aspect of the experiment design was closely tied to the constraints of the continuous rating paradigm. This limitation is particularly relevant for the analysis of the acoustic and musical features contributing to feeling moved, since the pattern of results may be specific to the particular music excerpts. It is likely that the acoustic and musical characteristics associated with feeling moved vary somewhat from piece to piece, and thus our findings reflect only a part of the entire picture. Moreover, the musical features that were explored in the present study do not represent an exhaustive selection of possible musical features, and thus there may be other, additional musical or acoustic features that contribute to feeling moved or touched. It is also likely that the relationship between musical features and perceptual ratings is non-linear, limiting the explanatory power of linear analysis methods such as those used in the present study. Similarly, we observed higher heterogeneity in effects for the sadly moving excerpts in comparison to the happily moving excerpts. This was mostly driven by Oblivion that typically showed smaller effects than the other two sadly moving excerpts, but sometimes even the opposite as for example for the relationship between feeling moved or touched and warmth in the chest. It is unclear whether that particular excerpt represents a more prototypical or more atypical version of the category of sadly moving music. Future studies employing a larger stimulus pool could remedy this shortcoming. The generalizability of the current findings hinges not only on the limited stimulus pool, but also on the employed sample. Amazon MTurk participants have been repeatedly criticized for providing low-quality data because they do not invest sufficient effort, are non-naive, and less trustworthy (e.g., [98]). However, the evidence for these claims seems to be rather mixed and dependent on specific situations [99]. Hauser et al. [99] highlight that high-quality data can be collected from MTurk, given specific research designs and an increased focus on attention checks. In the present study, we employed several attention checks and tried to make sure that participants understood the main task in order to avoid the possible pitfalls associated with MTurk participants. The fact that we replicated the main findings from a previous study using a similar paradigm [53] also speaks to the validity of our results. Nevertheless, we acknowledge that conclusions beyond the present sample and also regarding possible cross-cultural differences in being moved responses (see [5]) are necessarily limited. However, our findings are an important first step towards understanding the contributing musical features, as well as the extent to which these may vary between sadly and joyfully moving music. Future studies could investigate the contributing features in a larger and more varied set of music examples and more diverse cross-cultural populations. A possibility for targeting such larger stimuli sets could be to instruct participants to self-select moving music (e.g., [43, 100]). While such a design minimizes the control over the stimulus material, it would be interesting to see whether the current findings replicate when employing such a technique. Furthermore, other types of analyses and experiments might shed more light on the musical cues and appraisals that are associated with feeling moved. For example, is feeling moved dependent on experiencing music as conveying prosocial intentions, or perceiving musical events in terms of prosocial interactions between agents (cf. [8])? Another limitation is related to the continuous rating paradigm itself, where the participants are asked to continuously monitor their subjective experience and adjust their ratings accordingly. It may be that the additional cognitive load of monitoring one’s internal states might detract from the intensity of the experience itself. We tried to mitigate the potential effects of this possibility by having participants only use one continuous scale per excerpt, and by using a Likert-type scale with 5 steps rather than a visual analogue slider, for example, making it less likely for participants to have to constantly adjust their rating. A complementary approach for future investigations could be to measure psychophysiological responses with and without continuous ratings, for example, in order to obtain an additional index of emotional arousal. This could also help with identifying peaks in physiological arousal, and whether these coincide with self-reported emotional peaks. Based on our design we were not able to assess intra-individual processes, as participants never rated more than one scale for the same musical piece. All presented comparisons were made on an inter-individual level. While it is intriguing to see the amount of convergence for averaged ratings of different individuals for the same musical excerpt, this approach might have overestimated correlations among the different ratings, as within error variance remains undetected. We also included two musical excerpts that featured vocals. While none of the participants were proficient in the depicted languages, the human voice is known to transport specific emotions as well as empathy (e.g., [101]). Nevertheless, the two vocal pieces showed similar results as the instrumental excerpts. In addition, we assessed perceived joy and sadness in the music, and not experienced joy and sadness induced by the music. Previous studies [7] and our pilot study suggested that ratings of perceived and experienced emotions in music correlate highly. Nonetheless, it should be emphasized that our measures only allow for indirect inferences about participants’ experiences of joy and sadness. Finally, in contrast to previous studies employing dichotomous ratings (e.g., [43]) we measured subjective chills using a continuous scale (based on Schubert et al. [53]). This was done in order to assess different intensities of chills or goosebumps. Importantly, we might therefore consider responses that would not be classified as chills responses with dichotomous rating paradigms. This is also obvious in the rather long average duration of chills ratings above the lowest scale point, suggesting that we not only covered peak-states but also lower intensities of subjective chills. Overall, we replicated associations between chills and different variables such as feeling moved, pointing to the validity of this measure. Notably, previous studies also differ in the reported average length of chills responses [43, 102], which is probably related to different methodologies and the exact operationalization of the phenomenon (i.e., as chills, goosebumps, or thrills). It is possible that for example thrills refer to a higher intensity response than goosebumps or that several short chill episodes occurring in quick succession are perceived as one longer response. A more standardized definition of the phenomenon of chills (see [44]) would be helpful in comparing different findings and evaluating the degree to which they investigate the same phenomenon. Conclusion In sum, the findings of this study demonstrate that musically evoked experiences of feeling moved are associated with a similar pattern of appraisals, physiological sensations, and trait correlations as feeling moved by videos depicting social scenarios. The observed pattern of components is consistent with the predictions of different theories on feeling moved, specifically kama muta theory. Feeling moved or touched by both sadly and joyfully moving music was associated with experiencing a sense of connection and perceiving joy in the music, while perceived sadness was associated with feeling moved or touched only in the case of sadly moving music. Acoustic features related to arousal contributed to feeling moved only in the case of joyfully moving music. Finally, trait empathic concern was positively associated with feeling moved or touched by music. These findings support the role of social cognitive and empathic process in music listening, and highlight the social-relational aspects of feeling moved or touched by music. Supporting information S1 File. Supplementary material. Supplementary information, including S1–S9 Tables and S1–S9 Figs. https://doi.org/10.1371/journal.pone.0261151.s001 (DOCX)

Brochot, Etienne;Souplet, Vianney;Follet, Pauline;Ponthieu, Pauline;Olivier, Christophe;Even, Gaël;Audebert, Christophe;Malbec, Rémi

doi: 10.1371/journal.pone.0262311pmid: 35025936

1. Introduction Since its first detection in Wuhan (China) in December 2019, the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) has rapidly spread to reach other countries worldwide as the coronavirus 2019 disease (COVID-19) became pandemic [1]. The virion has a nucleocapsid composed by genomic RNA and phosphorylated Nucleocapsid (NP) protein, which is buried inside a phospholipid bilayer and covered by the Spike proteins trimmers (S) that gives the CoVs their crown-like appearance on which their names are based. The S protein has two subunits, the Spike 1 (S1) which contains the receptor-binding domain (RBD) and N-terminal domain (NTD) and the Spike 2 (S2) [2]. The choice of the antigenic domain is important, as it must be specific to the SARS-CoV-2 for discrimination against other hCoVs for example, and sensitive enough so infection would not be missed [3]. Also, anti-RBD antibodies are known to play a role in patients protection as this domain is used by the virus to penetrate host cells [4]. Most commercial serological assays have demonstrated satisfying performances in terms of diagnostic sensitivity and specificity, based on one of those main different antigenic domains [5, 6]. However, the combination of different immunogenic antigens can give a more comprehensive picture of the humoral response strength and diversity [7–9] while maintaining elevated diagnostic performances [10, 11]. In multiplex assays, positivity thresholds can be adjusted to compensate for the use of antigenic domains more conserved between coronaviruses [12]. Moreover, as vaccines are based on the Spike protein, the additional detection of anti-NP antibodies allows to differentiate viral infection from vaccination. This study reports the use of the CoViDiag® multiplex IgG assay for the characterization of the immune response against over time, depending on disease severity, and in perspective of neutralizing antibody titers. 2. Material and methods 2.1. Study design and cohort The study was conducted at Amiens University medical Center (France). Samples were derived from de-identified excess serum specimens. The demographic information of the patients are available in Table 1. The study was approved by the institutional review board of the Amiens University Medical Center (number PI2020_843_0046, 21 April 2020). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Cohort characteristics. https://doi.org/10.1371/journal.pone.0262311.t001 Briefly, we used n = 209 samples collected between March and April 2020 from n = 61 patients (27 hospitalized patients and 34 non-hospitalized patients) with PCR-confirmed SARS-CoV-2 infections to perform immunoassay and virus seroneutralization test as already described in Aubry et al. [13]. All samples have been tested according to manufacturer’s instruction on the CoViDiag® serological assay and the raw results are available in supplementary data. 2.2. Serological assay The CoViDiag® multiplex immunoassay is based on the ELISA principle and targets IgG antibodies against five different antigens of the SARS-CoV-2 virus: NP, S1, S2, RBD, and NTD (Fig 1). Note that the S1 and NP antigens have been printed in dot replicates in the shape of an “S” and “N” letters, respectively. This design allows for quick visual interpretation of seropositivity and vaccination status according to the manufacturer’s instruction (IFU in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Full well pictures pictures obtained with the microplate reader (SciReader®) or with a phone camera (in insert) after incubation with the CoViDiag® assay. (A) Positive sample presenting antibodies against the Nucleopcapside (NP), Spike 1 (S1), N-terminal domain (NTD) and Receptor binding domain (RBD) of the Spike protein, or Spike 2 (S2) antigens. (B) Negative sample with positive control on the edges. Scale bars correspond to 1 mm. https://doi.org/10.1371/journal.pone.0262311.g001 Briefly, serum samples (100 μL per well) were diluted 1:100 in the provided ready to use Diluent Buffer. The plates were incubated 1 h at 37 °C on a microplate shaker at 300 rpm, and washed three times (200 μL/well) with the provided Washing Buffer. 60 μL of Conjugate Antibody was added to 10 mL of Dilution Buffer for conjugation and 100 μL of diluted conjugate was added to each well, followed by 1 h incubation at 37 °C in the dark. After washing, 50 μL of provided Substrate solution was added to each well and incubated for 15 min in the dark. After a final washing step with 200 μL of mQ water per well, any trace of residual water was removed by incubation for 15 min at 37 °C. Distinguishable individual spot (circular “blue dots”) are visible at the surface of the wells when IgG antibodies have been specifically captured by the corresponding antigens. The color intensity is correlated to the amount of antibodies present in the sample. Images of individual wells were captured by a microplate reader (SciReader®, Scenion GmbH) and associated software for spot detection and spot intensity measurement. The spot mean signal intensity (MSI) in arbitrary unit (a.u.) was calculated as the average pixel value inside the spot perimeter minus the local background around the spot as described in Malbec et al. [14]. For automatic delivering of the diagnosis results, an algorithm combining different cut-off (reported in Table 2) has been set in the software as recommended by the CoViDiag® Instruction For Use (see IFU section 8.4 in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Mean signal intensity cut-offs for individual antigens in arbitrary units (a.u.). https://doi.org/10.1371/journal.pone.0262311.t002 Samples are identified as IgG positive to SARS-CoV-2 when S1 and/or RBD and/or NTD is positive, or S2 and/or NP MSI is > 40 a.u, or S2 and/or NP is positive and S1 and/or RBD and/or NTD and/or S2 and/or NP is borderline, or S2 and/or NP is borderline and S1 and/or RBD and/or NTD and/or S2 and/or N is borderline. 2.3. Neutralization assay Retroviral particles pseudotyped with the S glycoprotein of SARS-CoV-2 (SARS-CoV-2pp) were produced, with a plasmid encoding a human codonoptimized sequence of the SARS-CoV-2 spike glycoprotein (accession number: MN908947), as previously described in Brochot et al. [3]. Supernatants containing the pseudotyped particles were harvested at 48, 72, and 96 h after transfection, pooled, and filtered through 0.45-μm pore-sized membranes. Neutralization assays were performed by preincubating SARS-CoV-2pp and serially diluted plasma for 1 h at room temperature before contact with 293T cells (ATCC® CRL-3216TM) transiently transfected with the plasmids pcDNA3.1-hACE2 24 h before inoculation. Luciferase activity was measured 72 h postinfection, as indicated by the manufacturer (Promega). Two independent tests were carried out each time in duplicate. The NAb titers were defined as the highest dilution of plasma resulting in a 90% decrease in infectivity. We previously controlled the specificity of our neutralization assay using not only plasmas from patients seropositive for other coronaviruses but also retroviral particles pseudotyped with the G glycoprotein of the vesicular stomatitis virus. 2.4. Statistical analysis For the statistical analysis, Student’s test was used to test the relationship between different categorical variables and the difference in antibody MSI between hospitalized and non-hospitalized groups of patients. Spearman’s rank Correlation test was used to test the correlation between different antibody MSI and dilution factor for the neutralization assay. The general significance level was set at a p-value below 0.05. All analyses were performed using packages stats from the R statistical computing program v. 3.6.1 (Date of release 07/05/2019). 2.1. Study design and cohort The study was conducted at Amiens University medical Center (France). Samples were derived from de-identified excess serum specimens. The demographic information of the patients are available in Table 1. The study was approved by the institutional review board of the Amiens University Medical Center (number PI2020_843_0046, 21 April 2020). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Cohort characteristics. https://doi.org/10.1371/journal.pone.0262311.t001 Briefly, we used n = 209 samples collected between March and April 2020 from n = 61 patients (27 hospitalized patients and 34 non-hospitalized patients) with PCR-confirmed SARS-CoV-2 infections to perform immunoassay and virus seroneutralization test as already described in Aubry et al. [13]. All samples have been tested according to manufacturer’s instruction on the CoViDiag® serological assay and the raw results are available in supplementary data. 2.2. Serological assay The CoViDiag® multiplex immunoassay is based on the ELISA principle and targets IgG antibodies against five different antigens of the SARS-CoV-2 virus: NP, S1, S2, RBD, and NTD (Fig 1). Note that the S1 and NP antigens have been printed in dot replicates in the shape of an “S” and “N” letters, respectively. This design allows for quick visual interpretation of seropositivity and vaccination status according to the manufacturer’s instruction (IFU in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Full well pictures pictures obtained with the microplate reader (SciReader®) or with a phone camera (in insert) after incubation with the CoViDiag® assay. (A) Positive sample presenting antibodies against the Nucleopcapside (NP), Spike 1 (S1), N-terminal domain (NTD) and Receptor binding domain (RBD) of the Spike protein, or Spike 2 (S2) antigens. (B) Negative sample with positive control on the edges. Scale bars correspond to 1 mm. https://doi.org/10.1371/journal.pone.0262311.g001 Briefly, serum samples (100 μL per well) were diluted 1:100 in the provided ready to use Diluent Buffer. The plates were incubated 1 h at 37 °C on a microplate shaker at 300 rpm, and washed three times (200 μL/well) with the provided Washing Buffer. 60 μL of Conjugate Antibody was added to 10 mL of Dilution Buffer for conjugation and 100 μL of diluted conjugate was added to each well, followed by 1 h incubation at 37 °C in the dark. After washing, 50 μL of provided Substrate solution was added to each well and incubated for 15 min in the dark. After a final washing step with 200 μL of mQ water per well, any trace of residual water was removed by incubation for 15 min at 37 °C. Distinguishable individual spot (circular “blue dots”) are visible at the surface of the wells when IgG antibodies have been specifically captured by the corresponding antigens. The color intensity is correlated to the amount of antibodies present in the sample. Images of individual wells were captured by a microplate reader (SciReader®, Scenion GmbH) and associated software for spot detection and spot intensity measurement. The spot mean signal intensity (MSI) in arbitrary unit (a.u.) was calculated as the average pixel value inside the spot perimeter minus the local background around the spot as described in Malbec et al. [14]. For automatic delivering of the diagnosis results, an algorithm combining different cut-off (reported in Table 2) has been set in the software as recommended by the CoViDiag® Instruction For Use (see IFU section 8.4 in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Mean signal intensity cut-offs for individual antigens in arbitrary units (a.u.). https://doi.org/10.1371/journal.pone.0262311.t002 Samples are identified as IgG positive to SARS-CoV-2 when S1 and/or RBD and/or NTD is positive, or S2 and/or NP MSI is > 40 a.u, or S2 and/or NP is positive and S1 and/or RBD and/or NTD and/or S2 and/or NP is borderline, or S2 and/or NP is borderline and S1 and/or RBD and/or NTD and/or S2 and/or N is borderline. 2.3. Neutralization assay Retroviral particles pseudotyped with the S glycoprotein of SARS-CoV-2 (SARS-CoV-2pp) were produced, with a plasmid encoding a human codonoptimized sequence of the SARS-CoV-2 spike glycoprotein (accession number: MN908947), as previously described in Brochot et al. [3]. Supernatants containing the pseudotyped particles were harvested at 48, 72, and 96 h after transfection, pooled, and filtered through 0.45-μm pore-sized membranes. Neutralization assays were performed by preincubating SARS-CoV-2pp and serially diluted plasma for 1 h at room temperature before contact with 293T cells (ATCC® CRL-3216TM) transiently transfected with the plasmids pcDNA3.1-hACE2 24 h before inoculation. Luciferase activity was measured 72 h postinfection, as indicated by the manufacturer (Promega). Two independent tests were carried out each time in duplicate. The NAb titers were defined as the highest dilution of plasma resulting in a 90% decrease in infectivity. We previously controlled the specificity of our neutralization assay using not only plasmas from patients seropositive for other coronaviruses but also retroviral particles pseudotyped with the G glycoprotein of the vesicular stomatitis virus. 2.4. Statistical analysis For the statistical analysis, Student’s test was used to test the relationship between different categorical variables and the difference in antibody MSI between hospitalized and non-hospitalized groups of patients. Spearman’s rank Correlation test was used to test the correlation between different antibody MSI and dilution factor for the neutralization assay. The general significance level was set at a p-value below 0.05. All analyses were performed using packages stats from the R statistical computing program v. 3.6.1 (Date of release 07/05/2019). 3. Results 3.1. Evolution of the IgG profile over time Using the CoViDiag® assay on 209 serum samples, we have observed over 87% seropositivity up to 6 months after an initial positive SARS-CoV-2 RT–PCR, before decreasing to 83% between six and eight months, as the seropositivity of individual IgGs targeting different antigens starts to drop (Fig 2A). As the overall IgG response gets weaker in time, the combined detection of IgGs to different antigenic domain allow to maintain elevated diagnostic sensitivity. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Evolution of the IgG profile over time. (A) Percentage of patients CoViDiag® positive to anti-NP, anti-S1, anti-S2, anti-RBD, and anti-NTD IgG antibodies and (B) associated average spot intensity (MSI) of IgG responses expressed as arbitrary units (a.u.) over time. https://doi.org/10.1371/journal.pone.0262311.g002 Positivities for each IgG considered individually are also reported based on the cut-offs set by the manufacturer (Table 2). More than a half of the samples (54.1%, n = 113/209) were concomitantly positives for anti-NP, anti-S1, anti-S2 and anti-RBD antibodies and 9.1% (n = 19/209) for all 5 antibodies. This result show that infected people generally develop antibodies against a wide spectra of the virus immunogenic domains. However, 4.3% (n = 9/209) samples (n = 6 for anti-NP, n = 2 for anti-S1 and n = 1 for anti-S2) presented a single positive antibody against the tested immunogenic domains (Table 3). The combination of multiple antigens may then help to slightly increase diagnostic sensitivity. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Prevalence of the profile of IgG immune response. Number of samples with antibodies targeting single or combination of antigenic domains. https://doi.org/10.1371/journal.pone.0262311.t003 The kinetics of the IgG serum antibody response to individual antigens are presented in Fig 2B. Average MSIs have been calculated for all samples depending on the time post RT-PCR to SARS-CoV-2. The anti-NP and anti-NTD antibody responses were the first to decrease, as their MSI started to decline after just two months (-0.9% and -8.1% between two and four months, respectively). The anti-S1 and anti-RBD response peaked after four months, before significatively decreasing over time (-7.8% and -13% between four and six months, respectively). The anti-S2 antibody response was the most delayed, with a peak level reached between four and six months. The different dynamics observed are in accordance with the combination of multi-antigens at different time point. In the first two months after a positive RT-PCR to SARS-CoV-2, an IgG response to a single antigen is observed in 5.8% (n = 3/52) of the samples and a concomitant IgG response to NP, S1, S2, and RBD antigens is observed in 51.9% (n = 27/52) of the samples (S1 Table). Between two and six months, the increase of the MSI measured for the different IgGs correlates with a diversification of the IgG response, as the IgG response to a single antigen is only observed in 2% (n = 2/98) of the samples while the frequency of observation of concomitant IgG response to NP, S1, S2, and RBD antigens increases to 61.2% (n = 60/98) of the samples. However after six months, the diversity of the IgG response decrease with the measured MSI, and IgG response to a single antigen is observed in 6.8% (n = 4/59) of the samples while the frequency of observation of concomitant IgG response to NP, S1, S2, and RBD antigens drops to 39% (n = 23/59) of the samples. Those results show the interest of detecting IgG response against multiple immunogenic domains to maintain elevated diagnostic sensitivity, especially long after infection. 3.2. IgG profile depending on the disease severity Then we have investigated the ability for the multiplex assay to differentiate hospitalized (severe cases) versus non hospitalized (mild cases) patients, based on the first sample collected for each of the 61 patients in the early convalescent phase of the disease. For all five immunogenic domains, the MSI, corresponding to the levels of antibody are plotted in Fig 3, depending on disease severity. For each given antigen, we have observed a trend of greater antibody response for hospitalized patients (MSI: NP = 56.5 a.u.; S1 = 49.1 a.u.; S2 = 59.4 a.u.; RBD = 54.8 a.u.; NTD = 11.8 a.u.; Average = 46.3 a.u.) compared to non-hospitalized ones (MSI: NP = 51.8 a.u.; S1 = 37.4 a.u.; S2 = 49.2 a.u.; RBD = 47.1 a.u.; NTD = 4.3 a.u.; Average = 37.9 a.u.). However, the differences were not statistically different (p-value > 0.05, S2 Table). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. IgG profile depending on disease severity outcome. Distribution of the different IgG responses based on the MSI in arbitrary units (a.u.), considered individually, or altogether (average) for hospitalized (n = 25) and non-hospitalized patients (n = 34) just after infection. https://doi.org/10.1371/journal.pone.0262311.g003 3.3. Correlation between IgG profiles and neutralizing antibody titers Finally, we have evaluated the ability for the correlation between the different IgG levels response and the seroneutralization potential of the samples. For all five immunogenic domains, the mean intensity, corresponding to the levels of antibody response are plotted in Fig 4 depending on the highest dilution of serum resulting in a 90% decrease in infectivity. As expected, the best correlation (see S3 Table) between individual IgGs and neutralizing antibody response was obtained for anti-RBD antibodies (r2 = 0.72, p-value < 2.2e-16). The correlation was very similar between anti-S1 (r2 = 0.67, p-value < 2.2e-16) and anti-S2 (r2 = 0.66, p-value < 2.2e-16) antibodies. However Anti-NP (r2 = 0.59, p-value < 2.2e-16) and anti-NTD (r2 = 0.47, p-value = 3.813e-13) antibodies MSI were less correlated with the neutralizing antibody titers. Interestingly, the combination of the 5 different antibody responses, allowed to slightly increase the correlation to (r2 = 0.74). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. Correlation of the different IgG responses with serum neutralization titers. IgGs responses are based on the MSI in arbitrary units (a.u.) considered individually, or altogether (average). Neutralizing antibody titers are based on the serum dilution factor to neutralize 90% of infected cells. https://doi.org/10.1371/journal.pone.0262311.g004 3.1. Evolution of the IgG profile over time Using the CoViDiag® assay on 209 serum samples, we have observed over 87% seropositivity up to 6 months after an initial positive SARS-CoV-2 RT–PCR, before decreasing to 83% between six and eight months, as the seropositivity of individual IgGs targeting different antigens starts to drop (Fig 2A). As the overall IgG response gets weaker in time, the combined detection of IgGs to different antigenic domain allow to maintain elevated diagnostic sensitivity. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Evolution of the IgG profile over time. (A) Percentage of patients CoViDiag® positive to anti-NP, anti-S1, anti-S2, anti-RBD, and anti-NTD IgG antibodies and (B) associated average spot intensity (MSI) of IgG responses expressed as arbitrary units (a.u.) over time. https://doi.org/10.1371/journal.pone.0262311.g002 Positivities for each IgG considered individually are also reported based on the cut-offs set by the manufacturer (Table 2). More than a half of the samples (54.1%, n = 113/209) were concomitantly positives for anti-NP, anti-S1, anti-S2 and anti-RBD antibodies and 9.1% (n = 19/209) for all 5 antibodies. This result show that infected people generally develop antibodies against a wide spectra of the virus immunogenic domains. However, 4.3% (n = 9/209) samples (n = 6 for anti-NP, n = 2 for anti-S1 and n = 1 for anti-S2) presented a single positive antibody against the tested immunogenic domains (Table 3). The combination of multiple antigens may then help to slightly increase diagnostic sensitivity. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Prevalence of the profile of IgG immune response. Number of samples with antibodies targeting single or combination of antigenic domains. https://doi.org/10.1371/journal.pone.0262311.t003 The kinetics of the IgG serum antibody response to individual antigens are presented in Fig 2B. Average MSIs have been calculated for all samples depending on the time post RT-PCR to SARS-CoV-2. The anti-NP and anti-NTD antibody responses were the first to decrease, as their MSI started to decline after just two months (-0.9% and -8.1% between two and four months, respectively). The anti-S1 and anti-RBD response peaked after four months, before significatively decreasing over time (-7.8% and -13% between four and six months, respectively). The anti-S2 antibody response was the most delayed, with a peak level reached between four and six months. The different dynamics observed are in accordance with the combination of multi-antigens at different time point. In the first two months after a positive RT-PCR to SARS-CoV-2, an IgG response to a single antigen is observed in 5.8% (n = 3/52) of the samples and a concomitant IgG response to NP, S1, S2, and RBD antigens is observed in 51.9% (n = 27/52) of the samples (S1 Table). Between two and six months, the increase of the MSI measured for the different IgGs correlates with a diversification of the IgG response, as the IgG response to a single antigen is only observed in 2% (n = 2/98) of the samples while the frequency of observation of concomitant IgG response to NP, S1, S2, and RBD antigens increases to 61.2% (n = 60/98) of the samples. However after six months, the diversity of the IgG response decrease with the measured MSI, and IgG response to a single antigen is observed in 6.8% (n = 4/59) of the samples while the frequency of observation of concomitant IgG response to NP, S1, S2, and RBD antigens drops to 39% (n = 23/59) of the samples. Those results show the interest of detecting IgG response against multiple immunogenic domains to maintain elevated diagnostic sensitivity, especially long after infection. 3.2. IgG profile depending on the disease severity Then we have investigated the ability for the multiplex assay to differentiate hospitalized (severe cases) versus non hospitalized (mild cases) patients, based on the first sample collected for each of the 61 patients in the early convalescent phase of the disease. For all five immunogenic domains, the MSI, corresponding to the levels of antibody are plotted in Fig 3, depending on disease severity. For each given antigen, we have observed a trend of greater antibody response for hospitalized patients (MSI: NP = 56.5 a.u.; S1 = 49.1 a.u.; S2 = 59.4 a.u.; RBD = 54.8 a.u.; NTD = 11.8 a.u.; Average = 46.3 a.u.) compared to non-hospitalized ones (MSI: NP = 51.8 a.u.; S1 = 37.4 a.u.; S2 = 49.2 a.u.; RBD = 47.1 a.u.; NTD = 4.3 a.u.; Average = 37.9 a.u.). However, the differences were not statistically different (p-value > 0.05, S2 Table). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. IgG profile depending on disease severity outcome. Distribution of the different IgG responses based on the MSI in arbitrary units (a.u.), considered individually, or altogether (average) for hospitalized (n = 25) and non-hospitalized patients (n = 34) just after infection. https://doi.org/10.1371/journal.pone.0262311.g003 3.3. Correlation between IgG profiles and neutralizing antibody titers Finally, we have evaluated the ability for the correlation between the different IgG levels response and the seroneutralization potential of the samples. For all five immunogenic domains, the mean intensity, corresponding to the levels of antibody response are plotted in Fig 4 depending on the highest dilution of serum resulting in a 90% decrease in infectivity. As expected, the best correlation (see S3 Table) between individual IgGs and neutralizing antibody response was obtained for anti-RBD antibodies (r2 = 0.72, p-value < 2.2e-16). The correlation was very similar between anti-S1 (r2 = 0.67, p-value < 2.2e-16) and anti-S2 (r2 = 0.66, p-value < 2.2e-16) antibodies. However Anti-NP (r2 = 0.59, p-value < 2.2e-16) and anti-NTD (r2 = 0.47, p-value = 3.813e-13) antibodies MSI were less correlated with the neutralizing antibody titers. Interestingly, the combination of the 5 different antibody responses, allowed to slightly increase the correlation to (r2 = 0.74). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. Correlation of the different IgG responses with serum neutralization titers. IgGs responses are based on the MSI in arbitrary units (a.u.) considered individually, or altogether (average). Neutralizing antibody titers are based on the serum dilution factor to neutralize 90% of infected cells. https://doi.org/10.1371/journal.pone.0262311.g004 4. Discussion Several studies have found improved performances from use of antigen combinations that include both spike protein and Nucleoprotein [10, 15, 16]. Gillot et al. evaluated the CoviDiag® assay and concluded that the combination of several antigens in the same test improves the overall specificity and sensitivity of the test [17]. Similarly, in our previous work based on the same set of sample, we have found equivalent to improved diagnostic performances, especially for ancient infections, for the CoViDiag® multiplex IgG assay compared to other simplex IgG commercial assays [13]. Is is now generally admitted that antibody levels are weaker for asymptomatic and mild form of the disease and can decrease over time. For instance, Grossberg et al., have observed a more robust IgG response in positive/symptomatic participants than in positive/asymptomatic participants [8]. They were able to differentiate between severe, mild and asymptomatic group of participants using S1-RBD IgA, NP IgG and S2 IgA titers. Hence in the present work, we have investigated the profile of the IgG immune response over an eight months period with a multiplex assay, using samples of hospitalized and non-hospitalized patients. Then we have compared the results with neutralizing antibody levels. We have observed that most patients develop a global immune response against multiple immunogenic domains. Even over an eight months period, more than a half of the samples were positives to anti-NP, anti-S1, anti-S2, and anti-RBD antibodies, concomitantly. Those result confirm the possibility to develop serological assays based on different antigens. Anti-NTD antibodies are more scarce. Using the multiplex technology from Meso Scale Diagnostics, LLC, Chaudhury et al, have also observed that IgM and IgG antibodies were less reactive to NTD than NP or RBD antigens [18]. One explanation might come from the fact that this domain shows the lowest sequence identity compared to SARS-CoV Spike protein. So the IgG response to NTD antigen may be more naïve than for others, resulting in decrease sensitivity but increase specificity potential for diagnostics, which was the initial reason for its presence in the CoViDiag® multiplex assay. Also, as most SARS-CoV-2 infected patients develop antibodies against the NP antigen differentiation of infection from vaccination may be possible based on this antigen as vaccines are based on the Spike protein. As expected, the different IgGs responses decreased over time, but with different dynamics. As the overall IgG response gets weaker in time, the probability of detecting an IgG response to a single antigen increases. Hence, the detection of IgG response to different antigenic domain may allow to maintain elevated diagnostic sensitivity. The evolution of anti-S1 and anti-RBD responses is very similar, as RBD constitutes a domain of the Spike 1 protein. However, elevated levels of anti-S2 IgG seem to last longer. Therefore the detection of anti-S2 IgG may be of interest to maintain elevated diagnostic sensitivity longer after infection. However as the S2 domain is highly conserved among coronavirus, its presence may not be very specific to SARS-CoV-2 infection. The CoViDiag assay algorithm adapts the cut-offs depending on the number of different IgGs detected to deliver SARS-CoV-2 positivity status, and maintain diagnostic sensitivity and specificity performances over time. Those results may explain our previous observations on the same cohort [13], where we have observed that the CoViDiag® diagnostic sensitivity performance remained more stable over time than for two other commercial references of simplex IgG immunoassay (Abbot® and Euroimmun® IgG assays, based on the NP and the S1 antigen, respectively). For all the tested IgGs, we have found higher MSI for hospitalized patients than for non-hospitalized ones. However, the differences were not statistically significant as a large number of patients had no immune response detected for individual antigens, independently of the disease severity. Those results are in accordance with the finding of Gillot et al. using the CoViDiag® assay, who have observed a trend of higher signals for NP, S1, S2 and RBD antibodies from 14 days since symptom onset in critical patients, even if the differences were not statistically different compared to non-critical patients in their cohort [17]. It is noteworthy that most commercial assays outstanding performances have been established at the beginning of the epidemic, on samples with severe form of the disease, and possible strong immune response, as samples from hospitalized patients were the easier to collect. For people presenting a weaker immune response, multiplexing allows to test for extra domains that may help to slightly increase diagnostic sensitivity without compromising for diagnostic specificity. Except for anti-NTD antibodies, all different IgGs MSI were positively correlated with the neutralizing antibody titers. This result is not surprising considering our previous observation showing that anti-NP, anti-S1, anti-S2, and anti-RBD antibodies are concomitantly present in patient’s sera. As expected, the best correlation for individual antigen is obtained for antibodies targeting the virus RBD domain which is known to be involved in the penetration of the cells by the virus. However the average combination of all five antigens slightly increased the correlation, strengthening the interest for multiplexing. Even if testing for IgGs seem more appropriate for the evaluation of an efficient and long lasting protection of the patients, the restriction to this particular isotype is a limitation to this study. Several commercial assays have shown good performances focusing on the detection of total antibodies (IgG, IgM and IgA). Evaluating the IgA and IgM profile in multiplex would be of interest for future experiments. Future studies could also include the collection of samples with more uniform number and duration, the determination of antibody titers using calibration curves, and investigate the immune profile between more diverse forms of the disease as asymptomatic forms. However the present work contributes to provide insights into the dynamic and diversity of the immune response over time and depending on the disease severity. Our results confirms those of previous study on the potential for multiplexing to improve diagnostic performances of COVID-19 serology assays. 5. Conclusion Beyond the diagnosis of SARS-COV-2 infection, tools delivering a global picture of the patients’ immune response may also be of interest to improve the management and care of the patients and populations. Our results show that elevated IgGs responses against multiple viral epitope may be more characteristic of symptomatic patients, and correlates well with neutralizing antibodies. We recommend using assays targeting IgGs for the evaluation of a long lasting population protection and collective immunity. Furthermore, multiplexed assays have the potential to slightly increase diagnostic performances, especially for ancient or weak infections and be more representative of immune protection. For future epidemical studies, as the vaccination based on the Spike protein progresses, multiplex serological assays may also help to differentiate vaccination from viral infection and the immune response to different variants. Supporting information S1 Table. Prevalence of the profile of IgG immune response. Percentage of positivity for antibodies against different antigens or combination of antigens for different time period after RT-PCR positive to SARS-CoV-2. https://doi.org/10.1371/journal.pone.0262311.s001 (PPTX) S2 Table. Mean Signal Intensity (MSI) in arbitrary units (a.u.) of the antibody response to different antigens for hospitalized and non-hospitalized patients. https://doi.org/10.1371/journal.pone.0262311.s002 (PPTX) S3 Table. Correlation between individual antigen responses and neutralizing antibody titers. https://doi.org/10.1371/journal.pone.0262311.s003 (PPTX) S1 File. Instruction for use version 1.4 of the CoViDiag® assay. https://doi.org/10.1371/journal.pone.0262311.s004 (PDF)

Onukwugha, Franklin I.;Smith, Lesley;Kaseje, Dan;Wafula, Charles;Kaseje, Margaret;Orton, Bev;Hayter, Mark;Magadi, Monica

doi: 10.1371/journal.pone.0261973pmid: 35061757

Background mHealth innovations have been proposed as an effective solution to improving adolescent access to and use of Sexual and Reproductive Health (SRH) services; particularly in regions with deeply entrenched traditional social norms. However, research demonstrating the effectiveness and theoretical basis of the interventions is lacking. Aim Our aim was to describe mHealth intervention components, assesses their effectiveness, acceptability, and cost in improving adolescent’s uptake of SRH services in Sub-Saharan Africa (SSA). Methods This paper is based on a systematic review. Twenty bibliographic databases and repositories including MEDLINE, EMBASE, and CINAHL, were searched using pre-defined search terms. Of the 10, 990 records screened, only 10 studies met the inclusion criteria. The mERA checklist was used to critically assess the transparency and completeness in reporting of mHealth intervention studies. The behaviour change components of mHealth interventions were coded using the taxonomy of Behaviour Change Techniques (BCTs). The protocol was registered in the ‘International Prospective Register for Systematic Reviews’ (PROSPERO-CRD42020179051). Results The results showed that mHealth interventions were effective and improved adolescent’s uptake of SRH services across a wide range of services. The evidence was strongest for contraceptive use. Interventions with two-way interactive functions and more behaviour change techniques embedded in the interventions improved adolescent uptake of SRH services to greater extent. Findings suggest that mHealth interventions promoting prevention or treatment adherence for HIV for individuals at risk of or living with HIV are acceptable to adolescents, and are feasible to deliver in SSA. Limited data from two studies reported interventions were inexpensive, however, none of the studies evaluated cost-effectiveness. Conclusion There is a need to develop mHealth interventions tailored for adolescents which are theoretically informed and incorporate effective behaviour change techniques. Such interventions, if low cost, have the potential to be a cost-effective means to improve the sexual and reproductive health outcomes in SSA. Background Globally, adolescents and young people face enormous barriers accessing Sexual and Reproductive Health (SRH) information and services [1–3], especially in Sub-Saharan Africa (SSA) with a high burden of HIV/AIDS and unintended pregnancy [4–6]. These barriers such as lack of awareness of available services, lack of confidentiality, service providers attitude, social norms and values and restrictive policies, operate at different levels [5, 6]. Adolescent girls and young women accounted for 25% of all new HIV infections globally in 2017 and of all HIV infections occurring among adolescents in SSA; 80% are in girls aged 15–19 years [7]. Sub-Saharan Africa had the highest prevalence of adolescent pregnancy in the world, between 1995 and 2011, with an estimated 104 births per 1,000 women aged 15–19 [8, 9]; and young women aged 15–24 years account for 57% of abortions [10, 11]. With over 600 million mobile phone subscribers predicted by 2025 in SSA [12], representing about half of the population, mHealth innovations have been proposed as a solution to improving access to and use of health services among the underserved population, especially in settings with poor healthcare infrastructure [13, 14]. Mobile health or mHealth is defined as a medical and public health practice supported by mobile phones, tablets, patient monitoring devices, personal digital assistants (PDAs), and other wireless devices [14]. mHealth can offer timely, accurate and non-judgemental SRH information and services to adolescents [15]. A systematic review identified 487 mHealth programmes implemented in SSA between 2006 and 2016 [16], although few involved adolescent SRH. Furthermore, most programmes in the region have not been rigorously evaluated [17]. Previous reviews have collated and evaluated mHealth interventions to improve adolescent’s uptake of SRH services [18–20]. The reference to SRH in this paper is consistent with the United Nations definition: a state of complete physical, mental and social well-being in all matters relating to reproduction, enabling people to have a satisfying safe sex life and the freedom to decide if, when and how often to reproduce, which implies the right of men and women to be informed and to have access to safe, effective, affordable and acceptable methods of family planning of their choice [21]. While countries have expanded their vision of addressing people’s rights to a full and comprehensive range of SRH services (including reproductive cancers, gender-based violence, etc), we recognize that many developing countries, especially sub-Saharan Africa, are only able to offer a core package of basic SRH services [22]. Therefore, we focus on the core / basic SRH, including family planning/contraception, sexually transmitted infections (including HIV/AIDS), and pregnancy/termination-related issues. Smith and colleagues provided some evidence that interventions delivered by mobile phones improve contraception use, although none of the studies were carried out in SSA [18]. Evidence from a systematic review study showed that health promotion campaigns implemented with text messaging improved SRH knowledge, reduced unprotected sex, and increased STI testing among adolescents [19]. However, only three out of 35 studies in the review were based in low- and middle-income countries (LMICs). A more recent review which used SRH defined as access to comprehensive sexuality education; services to prevent, diagnose and treat STIs and counselling on family planning, and found mHealth interventions to be effective in improving uptake of antenatal care and postnatal care services, especially those that were aimed at changing behaviour of pregnant women [20]. However, the study reported paucity of evidence on other types of mHealth applications. Although these reviews shed some light on the quality of mHealth evidence, approaches, and barriers, in improving adolescents’ access to SRH services in LMICs, there is a dearth of understanding on the effectiveness of mHealth interventions in improving uptake of services specifically among adolescents in SSA. Our review builds on previous evidence by exploring the theoretical and empirical basis of mHealth interventions using a taxonomy of behaviour change techniques [23], and assesses the completeness of reporting mHealth interventions for improving adolescent’s uptake of SRH services using the WHO developed mERA checklist [24]. Research aim To determine the effectiveness of mHealth interventions to improve the uptake of sexual and reproductive health (SRH) services by adolescents in Sub-Saharan Africa. Primary objectives are to Describe the components of mHealth interventions addressing SRH among adolescents in SSA Assess the effectiveness of mHealth interventions addressing SRH among adolescents in SSA Secondary objectives are to assess the Acceptability of mHealth interventions to adolescents, and parents in providing SRH information in SSA. Feasibility of delivery of mHealth interventions by providers Cost-effectiveness of mHealth interventions in SSA. Research aim To determine the effectiveness of mHealth interventions to improve the uptake of sexual and reproductive health (SRH) services by adolescents in Sub-Saharan Africa. Primary objectives are to Describe the components of mHealth interventions addressing SRH among adolescents in SSA Assess the effectiveness of mHealth interventions addressing SRH among adolescents in SSA Secondary objectives are to assess the Acceptability of mHealth interventions to adolescents, and parents in providing SRH information in SSA. Feasibility of delivery of mHealth interventions by providers Cost-effectiveness of mHealth interventions in SSA. Methods This review was carried out using the guidance developed by the Centre for Reviews and Dissemination [25]. The protocol was registered in the ‘International Prospective Register for Systematic Reviews’ (PROSPERO) CRD42020179051 [26] The review was based on evaluation studies that assessed the effectiveness of mHealth interventions to support the delivery of information, decision-making, behaviour change or risk reduction strategies regarding SRH among adolescents aged 10–19 years. Although, our target population was adolescents aged 10–19 years [6], however, interventions that focus on young people aged 10–24 years was considered also as interventions that focus on young people aged 10–24 may likely not be different from those of adolescents aged 10–19 years. Inclusion and exclusion criteria This review was based on studies conducted in Sub-Saharan Africa (SSA) as defined by the UN Development Program 2020 [27]. We included any single or multi-component mHealth/mobile health interventions that supports delivery of information, decision-making, behaviour change or risk reduction strategies regarding Sexual and Reproductive Health (SRH). We included evaluation studies such as Randomized control trials (RCTs), other experimental and quasi/non-experimental studies that assess the effectiveness of mHealth interventions. Studies outside these parameters were not considered. Electronic searches Eight primary bibliographic databases (MEDLINE, EMBASE, CINAHL, PsycINFO, Web of Science, Cochrane Library Central Register of Controlled Trials, SCOPUS and Academic search premier), six institutional digital databases (WHO Global Health Library, African population and health research centre (APHRC), United Nations Population Fund (UNFPA), Guttmacher Institute, Population Council, and Family Health International) and other repositories (ProQuest, International Bibliography of social sciences, OpenDOAR, Ethos-British Library, Network digital library of Thesis and Dissertation and ZETOC) were searched from April to May 2020 for peer-reviewed articles and grey literature. There was no restriction in terms of language or publication year. Non-English language published papers on mHealth were not identified during our literature search. Search strategy The search strategy was developed by FO with input from LS, DK and MM. Search terms were iteratively developed within each of three search concepts: (i) Sexual and Reproductive Health; (ii) mHealth; (iii) Sub-Saharan Africa. The keywords and database thesaurus terms were combined one after the other using Boolean Operators and truncation/wildcards were applied and modified where appropriate. Full details of the review protocol is published online [26] and the full search strategy available as S1 File. Study records were exported to Endnote and titles and abstracts screened by FO and MK and disagreement resolved by MM. Full-text articles were independently screened for inclusion by three reviewers (FO, CW and MH) using Rayyan QCRI software and disagreement resolved by discussion with LS. Data extraction Data extracted were the author’s name and year, study design & sample size, study settings, interventions, target population and outcomes. Table 1 provides the full sample description. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Sample description. https://doi.org/10.1371/journal.pone.0261973.t001 Table 2 & Table 3 summarise the results of each paper. The WHO developed mHealth Evidence Reporting and Assessment (mERA) Checklist comprising 16 items focused on reporting mHealth interventions was used to critically assess the content, context, implementation features and completeness in reporting of mHealth studies [24]. Also, the behavioural change components of the mHealth interventions were coded using the taxonomy of Behavioural Change Techniques (BCTs) [23]. Data extraction was completed by FO and checked by LS. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Summary of results of included studies on mHealth interventions on SRH knowledge, sexual behaviour & contraceptive use. https://doi.org/10.1371/journal.pone.0261973.t002 Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Summary of results of included studies on mHealth interventions on ART, pregnancy & childbirth and breast feeding. https://doi.org/10.1371/journal.pone.0261973.t003 Risk of bias (quality) assessment Studies were appraised for methodological rigour/quality using the revised Cochrane risk-of-bias tool for randomized trials and the ROBINS-I tool for non-randomized studies [28]. Risk of bias assessment was completed by FO and independently reviewed by LS. RCTs were assessed based on random sequence generation (selection bias), allocation concealment, blinding, incomplete data, selective reporting, and other biases encountered throughout the study. The potential sources of bias for all the non-randomized studies were assessed based on the seven domains (selection of participants, measurement of interventions, departures from intended interventions, the control of cofounders, missing data, and selection of reported results) of the ROBINs-I tool. More details of the risk of bias for the studies are provided under the section ‘Characteristics of studies’. Strategy for data synthesis The results of the search were reported and presented in a Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) flow diagram. The extracted data for each included study was presented in tabular form, and results of the individual studies were narratively synthesised aligning with the review objectives as statistical pooling was not carried out due to the variation in study designs, interventions, sample population and outcome measures. See the review protocol for information on the strategy for data synthesis (CRD42020179051) [26]. Inclusion and exclusion criteria This review was based on studies conducted in Sub-Saharan Africa (SSA) as defined by the UN Development Program 2020 [27]. We included any single or multi-component mHealth/mobile health interventions that supports delivery of information, decision-making, behaviour change or risk reduction strategies regarding Sexual and Reproductive Health (SRH). We included evaluation studies such as Randomized control trials (RCTs), other experimental and quasi/non-experimental studies that assess the effectiveness of mHealth interventions. Studies outside these parameters were not considered. Electronic searches Eight primary bibliographic databases (MEDLINE, EMBASE, CINAHL, PsycINFO, Web of Science, Cochrane Library Central Register of Controlled Trials, SCOPUS and Academic search premier), six institutional digital databases (WHO Global Health Library, African population and health research centre (APHRC), United Nations Population Fund (UNFPA), Guttmacher Institute, Population Council, and Family Health International) and other repositories (ProQuest, International Bibliography of social sciences, OpenDOAR, Ethos-British Library, Network digital library of Thesis and Dissertation and ZETOC) were searched from April to May 2020 for peer-reviewed articles and grey literature. There was no restriction in terms of language or publication year. Non-English language published papers on mHealth were not identified during our literature search. Search strategy The search strategy was developed by FO with input from LS, DK and MM. Search terms were iteratively developed within each of three search concepts: (i) Sexual and Reproductive Health; (ii) mHealth; (iii) Sub-Saharan Africa. The keywords and database thesaurus terms were combined one after the other using Boolean Operators and truncation/wildcards were applied and modified where appropriate. Full details of the review protocol is published online [26] and the full search strategy available as S1 File. Study records were exported to Endnote and titles and abstracts screened by FO and MK and disagreement resolved by MM. Full-text articles were independently screened for inclusion by three reviewers (FO, CW and MH) using Rayyan QCRI software and disagreement resolved by discussion with LS. Data extraction Data extracted were the author’s name and year, study design & sample size, study settings, interventions, target population and outcomes. Table 1 provides the full sample description. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Sample description. https://doi.org/10.1371/journal.pone.0261973.t001 Table 2 & Table 3 summarise the results of each paper. The WHO developed mHealth Evidence Reporting and Assessment (mERA) Checklist comprising 16 items focused on reporting mHealth interventions was used to critically assess the content, context, implementation features and completeness in reporting of mHealth studies [24]. Also, the behavioural change components of the mHealth interventions were coded using the taxonomy of Behavioural Change Techniques (BCTs) [23]. Data extraction was completed by FO and checked by LS. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Summary of results of included studies on mHealth interventions on SRH knowledge, sexual behaviour & contraceptive use. https://doi.org/10.1371/journal.pone.0261973.t002 Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Summary of results of included studies on mHealth interventions on ART, pregnancy & childbirth and breast feeding. https://doi.org/10.1371/journal.pone.0261973.t003 Risk of bias (quality) assessment Studies were appraised for methodological rigour/quality using the revised Cochrane risk-of-bias tool for randomized trials and the ROBINS-I tool for non-randomized studies [28]. Risk of bias assessment was completed by FO and independently reviewed by LS. RCTs were assessed based on random sequence generation (selection bias), allocation concealment, blinding, incomplete data, selective reporting, and other biases encountered throughout the study. The potential sources of bias for all the non-randomized studies were assessed based on the seven domains (selection of participants, measurement of interventions, departures from intended interventions, the control of cofounders, missing data, and selection of reported results) of the ROBINs-I tool. More details of the risk of bias for the studies are provided under the section ‘Characteristics of studies’. Strategy for data synthesis The results of the search were reported and presented in a Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) flow diagram. The extracted data for each included study was presented in tabular form, and results of the individual studies were narratively synthesised aligning with the review objectives as statistical pooling was not carried out due to the variation in study designs, interventions, sample population and outcome measures. See the review protocol for information on the strategy for data synthesis (CRD42020179051) [26]. Results Study identification and selection The search identified 10,990 citations. After removing duplicates, 6,401 citations were included for title and abstract screening of which 86 full-text articles were assessed yielding 10 studies that met the review inclusion criteria. One RCT was published in two separate papers [29, 30], however, the main trial findings were reported in one [29] which was used throughout the review. See Fig 1 (PRISMA flow diagram). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. PRISMA flow diagram. https://doi.org/10.1371/journal.pone.0261973.g001 Study setting and design Four studies were carried out in Kenya [31–34], two in Ghana [29, 35], one in South Africa [36], two in Uganda [37, 38], and one in Tanzania [39]. Five studies were RCTs [29, 33, 34, 37, 38], three were evaluation and pre-post design studies [31, 32, 39], and two were mixed methods/cross-sectional studies [35, 36]. Seven studies occurred in a hospital setting [31–32, 36–38], one was school-based [29] and two were community-based [35, 39] (Table 1). Characteristics of studies All 5 RCTs were judged at low risk of bias for most criteria [29, 33, 34, 37, 38]. All reported random sequence generation except for one study [38], and all reported adequate method of concealment, losses to follow up and prespecified outcomes [29, 33, 34, 37, 38]. None of the RCTs reported blinding of participants. Almost all the five non-randomised studies showed a low risk of bias for selection of participants, measurement of interventions, departures from intended interventions. However, how confounding factors were controlled for and missing data were dealt with was not reported in any of the studies [31, 32, 35, 36, 39]. The measurement of outcomes in all the studies was based on a self-reported measure except for one study [37]. One of the non-randomised studies had significant technological challenges and inadequacies concerning the design of text message content, which could have influenced the outcome of the study [35]. Finally, only five studies out of ten analysed by intention-to-treat i.e. analysed the participants according to the groups to which they were originally assigned [29, 33, 34, 37, 38], which provides a more accurate, unbiased estimate of the findings reported in this study. mHealth interventions and platforms used. The most used mHealth platform was the Short Message Service (SMS) (9 studies). Only one of the studies used an interactive web-based peer support platform [31]. Five studies were based on unidirectional and interactive messaging services; three were based solely on 2-way interactive and two on unidirectional messaging services. The interventions focused on shaping knowledge and increasing the use of reproductive health interventions or services. Two studies evaluated SRH knowledge [29, 35], four assessed contraceptive use/birth control [29, 33, 34, 39], three examined pregnancy and fertility intentions [29, 33, 34]. One focused on facility childbirth delivery [33], two on exclusive breastfeeding (EBF) [33, 34], four on HIV Antiretroviral Therapy (ART) adherence [31, 32, 37, 38], and two on sexual behaviour [29, 34]. Effectiveness of the mHealth interventions on improving SRH outcomes SRH knowledge. Table 2 shows the effect of the intervention on SRH knowledge, sexual behaviour, and contraceptive use. Three studies examined adolescent SRH knowledge outcomes [29, 31, 35]. Only one study evaluating a unidirectional and interactive intervention among adolescent girls showed a positive effect at 3-months and 15 months after controlling for covariates, (age, religion, ethnicity, parents’ educational level [29]. Sexual and Reproductive Health (SRH) knowledge increased by 11% (95% Confidence Interval (CI): 7%, 15%) and 24% (95% CI: 19, 28), greater than in the control group, respectively. This effect was maintained at 15 months in the interactive group, however, ceased to be significant among the unidirectional group. The remaining two studies showed a difference in knowledge between the intervention and control groups [31, 35]. However, the intervention group in one of the studies reported more false answers on SRH knowledge about STIs, abortion & contraception (1.7) compared to the control group (1.9) [35]. Similarly, with reference to improvement in HIV/ART knowledge, there was no statistically significant change in knowledge among adolescents who participated in the interactive web-based intervention (adolescents demonstrated less knowledge at end-line comparing to baseline) and those that did not take part [31]. Sexual health behaviour. Two studies reported effects on adolescent’s sexual behaviour [29, 34]. One study found that the SMS intervention influenced young women’s resumption of sexual intercourse such that most women (31.8% at 6 weeks, 57.9% at 14 weeks, and 67.7% at 6 months) reported having resumed sexual intercourse by 6 months [34]. Similarly, in another study, the interactive intervention was positively associated with having sex without a condom among sexually active adolescents in the interactive group (OR = 3.47; 95% CI = 1.12, 10.74). However, the intervention did not influence the age of sexual debut for those who have ever had sexual intercourse [29]. (See Table 2). Contraceptive/Birth control access and use. Four studies reported on contraceptive use [29, 33, 34, 39]. Evidence from the four studies showed that the intervention increased the use and access to contraceptive services and family planning initiation among adolescents and this was higher among the interactive compared with unidirectional group (Table 2). One study found that highly effective contraceptive (HEC) use at 6 months postpartum was significantly higher among those in the SMS group (69.9%) than in the control group (57.4%) ([RR] = 1.22; 95%; 1.01, 1.47; P = .04) [32]. Another study reported that contraceptive use was significantly higher in both intervention arms by 16 weeks (1-way SMS: 72% and 2-way SMS: 73%; p = 0·03 and 0·02 versus 57% control, respectively) [33]. This trend was reported in another study which found that the interactive intervention increased the odds of using oral contraceptives (OR = 13.2; 95% CI = 1.08, 161) and decreased the odds of using emergency contraception (OR = 0.22; 95% CI = 0.05, 0.88) [29]. Likewise, in another study, participants who engaged with the intervention accessed contraceptive information more frequently than non-intervention group [39]. Antiretroviral therapy (ART) adherence. Table 2 presents the results of the effects of the interventions on antiretroviral therapy adherence (ART), pregnancy & childbirth and breast feeding. Four studies reported effects on ART adherence [31, 32, 37, 38]. Overall, the four studies showed an improvement in adherence and pre-exposure prophylaxis (PrEP) initiation. However, this improvement was not statistically significant except for one study [32]. Despite relying on a routine collected measure, an evaluative study found that women who enrolled in the intervention were almost twice more likely to continue PrEP (22% vs. 43%; aRR = 1.75; 95% CI = 1.21, 2.55; P = .003), than women who initiated PrEP in the month before the intervention implementation [32]. This is contradicted by another study which showed no statistical difference in adherence between the intervention and control groups (Adherence was 64% for the 1-way group [OR = 0.64; 95% CI: 0.58, 0.70; P = .27] and 61% for the 2-way group [0.56, 0.67; P = .15], compared with 67% in the control group (OR = 0.67; 95% CI:0.62, 0.72] [35]. Also, there was no statistically difference between the proportion of participants achieving adherence of at least 90% over the 48-week period of analysis (1-way group = 28%; 2-way group = 26% and control = 29%; P = .85 and .69, respectively). A similar study found that at baseline, 71.6% of participants reported not to have missed any doses in the last week, while 77.8% of the participants at the post intervention reported not to have missed any doses in the last week [31]. Although not statistically significant (p = 0.95) and finally, this level of insignificance persists in another study, which found that after controlling for baseline adherence, the intervention group 1 (T1) had 3.8% lower adherence than the control group (95% CI -9.9, 2.3) and the Intervention group 2 (T2) had 2.4 percentage points higher adherence than the control group (95% CI -3.0, 7.9). However, the differences were not statistically significant for either intervention groups [38]. Pregnancy and childbirth. Three studies reported the effect of the intervention on pregnancy and childbirth outcomes [29, 33, 34]. These RCTs studies showed that the intervention influenced fertility intentions, reduced the odds of self-reported pregnancy and facility delivery. However, the effects were not statistically significant between the intervention and the control groups except for one study [29], where both the unidirectional and the interactive groups significantly lowered the odds of self-reported pregnancy by 86% in the adjusted models (odds ratio [OR] = 0.14; 95% CI = 0.03, 0.71) and 85% (OR = 0.15; 95% CI = 0.03, 0.86), respectively, compared with the control group. In another study of 184 participants who initially wanted to become pregnant again and whose fertility intentions were similar, found that after 6-month visits, fertility intentions were similar between groups with 26.2% who reported a desire to stop childbearing [34]. A similar study stated that at 10 weeks, facility delivery was high in all 3 intervention arms [33]. Among 277 women providing delivery data, 273 (98.6%) reported delivering in a facility, with no difference between the 1-way and control arms [relative risk (RR) 1.00, 95% CI 0·97–1·03; p = 0·99] or 2-way and control arms [RR 0·99, 95% CI 0·95–1·03; p = 0·54]. Although there were apparently fewer still-births and infant deaths in the 2-way group compared to the control group (3·1% versus 8%), but not statistically significant (p = 0.21) [31]. No serious adverse events occurred because of the intervention although one maternal death occurred (See Table 3). Breastfeeding. As reported in Table 3, only two RCTs reported the effects of SMS intervention on exclusive breastfeeding (EBF) [33, 34]. Both studies reported inconsistent findings with one study showing a significant improvement in EBF among the intervention group than the control group [33], and another showing no significant difference across the two groups examined [34]. One of the studies revealed that women in both intervention arms were significantly more likely to report EBF at 10 and 16 weeks than women in the control arm [10 weeks: Control arm (RR: 0.79; CL:0.69-.86); 1-way SMS RR = 0.93 (CI:0.86–0.97; p = 0.003), 2-way SMS: RR = 0.96 (0.89–0.98; P = 0.0004); At 16 weeks: Control arm [RR = 0.62; CI: 0.52–0.71]; 1-way SMS [RR = 0.82; CI 0.72–0.89, P = 0.002], 2-way SMS [RR = 0.93, CI: 0.85–0.97; P = 0.001]. At 24 weeks, the probability of EBF was higher in both intervention groups than in the control, but only statistically significant in the 2-way messaging group [0·49 in 1-way, 0·62 in 2-way, and 0·41 in control, (p = 0·30 and 0·005 for 1-way and 2-way vs. control, respectively). Components and characteristics of mHealth interventions Behavioural change components of the interventions. Overall, 23 from a possible list of 93 BCTs were identified as intervention components in the included studies (Fig 2). The 23 BCTs were from six out of the 16 possible domains (feedback & monitoring, social support, shaping knowledge, natural consequences reward & threat) of BCTs [23]. The most commonly used BCTs in these studies were feedback & monitoring, and social support (6 studies). The feedback and monitoring techniques mostly used in the studies focused on monitoring and providing informative feedback on scores and performance of the behaviour among participants. However, the feedback was based on change in knowledge and not on change in behaviour. Half of the studies (five out of ten) described how participants were socially and practically supported to achieve the intervention objectives, although some of the studies did not specify the nature of social support provided, and four studies reported on shaping knowledge through instruction on how to perform a behaviour. Two interventions did not report the use of any BCTs [36, 39]. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Heat map: Showing the behavioural techniques used as intervention components in each study. Key: 1 = Goals & planning, 2 = Feedback & monitoring, 3 = Social support, 4 = Shaping knowledge, 5 = Natural consequences, 6 = Comparison of behaviour, 7 = Associations 8 = Repetition & substitution, 9 = Comparison of outcomes, 10 = Reward & threat, 11 = Regulation, 12 = Antecedents, 13 = Identity, 14 = Scheduled consequences. 15 = Self-belief, 16 = Covert learning. https://doi.org/10.1371/journal.pone.0261973.g002 The second aspect was to identify the mHealth intervention content for the included studies: where it is being implemented (context), and how it was implemented (technical features) to support replication of the intervention using the mHealth evidence reporting and assessment (mERA) guidelines [24]. Fig 3 below shows the number of included studies meeting each mHealth criterion. On average, about 35% (6%-63%) of the 16 mERA criteria was achieved among all the 10 studies. Overall, most studies described the mode and frequency of intervention delivery [29, 31–39], how people were informed of the programme [29, 31–39], how the content of the intervention was developed [29, 31–34, 36–39] and the technology platform/software used in the programme implementation [29, 31–34, 37–39]. However, there were limited information on the barriers/challenges faced by participants in adopting the intervention [36] study), the physical infrastructure used to support the interventions [32, 34] and the security and confidentiality protocol of the interventions [33, 34]. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Number of papers that met mERA essential criteria among the 10 selected studies. No of studies in each component: Infrastructure [32, 42]; Technology platform [29, 31–34, 37–39]; Interoperability [29, 31–34, 38]; Intervention delivery [29, 31–39]; Intervention content [29, 31–34, 36–39]; Usability testing [29, 31–35, 37]; User feedback [31, 32, 38, 39]; Access of individual participants [38]; Cost assessment [29, 34]; Adoption inputs/programme entry [29, 31–39]; Limitations for delivery at scale [29, 35–38]; Contextual adaptability [31–35, 38]; Replicability [29, 32–34, 37, 38]; Data security [33, 34]; Compliance [29, 32, 34, 36, 38, 39]; Fidelity of the intervention [29, 32–34, 37–39]. https://doi.org/10.1371/journal.pone.0261973.g003 Secondary outcomes Acceptability of mHealth interventions. Four studies evaluated adolescent acceptability of receiving an mHealth intervention for increasing adherence to HIV prevention or treatment [31, 32, 36, 38]. Three studies were among HIV positive adolescents [31, 36, 38] and one was pre-exposure prophylaxis among pregnant or post-partum women [32]. No study reported on the acceptability of mHealth interventions by parents of adolescents. Across all four of the studies, participants showed a positive attitude, and were willing to use and recommend mHealth interventions to others. One study reported that almost all the women (95%) would recommend the intervention (mWACh-PrEP) to other women who use pre-exposure prophylaxis (PrEP), and 95% would also use the program again if offered [32]. Also, 94% and 89% of women reported that SMS helped them understand and adhere to PrEP, respectively [32]. Another study found that among 81 adolescents with HIV that completed a mid-term questionnaire, the majority (95%) agreed to use the web-based interventions (ELIMIKA website) again and 87% would recommend it to others [31]. A related study in Uganda, showed that participants had positive attitudes about SMS as an incentive to adherence to ART (SITA) [38]. At follow-up, 96.6% of adolescents with HIV reported that they would remain in the intervention group if they had the choice (95.3% in the treatment 1 (T1) group and 97.8% in treatment 2 (T2) group, and 84.2% said there was nothing about SITA that they did not like (86.0% in T1 and 82.6% in T2). Participants from both intervention groups felt that SITA boosted their morale and prompted them to take their ART medication on time. Despite a small sample size, evidence from another study revealed that 65% of adolescents with HIV were willing to participate in a mHealth intervention to support treatment adherence [36]. Feasibility of delivery of mHealth interventions. Three studies reported on the feasibility of delivering mHealth intervention to adolescents in SSA [36, 38, 39]. All three of the studies established feasibility of delivering HIV treatment adherence and contraceptives information to adolescents using mHealth interventions. One study showed that sending text messages with information on a participant’s own adherence, information about the adherence performance of their peers and the recruitment process was practicable among HIV positive youth [38]. Also, a study found that use of text messages to support treatment adherence in adolescents with HIV was feasible, especially among in-school adolescents with high ownership of mobile phone with 67% willing to receive health related SMS [36]. Another study demonstrated that text messages comprising comprehensive information on contraceptive methods can be feasibly delivered and accessed by men and women of reproductive age [39]. However, Fig 3 (mERA checklist) above shows that only five studies reported the appropriateness of the intervention to the context and any possible adaptations required. Cost-effectiveness of mHealth interventions. None of the included studies reported cost-effectiveness outcomes. Intervention costs was among the least reported components in the mERA checklist (Fig 3). One study reported that the marginal costs of the interactive and unidirectional component per participant were US $1.91 and US $0.30, respectively [29]. Another study reported that the intervention was “relatively inexpensive “but with no information was provided on the specific costs of the intervention [34]. Study identification and selection The search identified 10,990 citations. After removing duplicates, 6,401 citations were included for title and abstract screening of which 86 full-text articles were assessed yielding 10 studies that met the review inclusion criteria. One RCT was published in two separate papers [29, 30], however, the main trial findings were reported in one [29] which was used throughout the review. See Fig 1 (PRISMA flow diagram). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. PRISMA flow diagram. https://doi.org/10.1371/journal.pone.0261973.g001 Study setting and design Four studies were carried out in Kenya [31–34], two in Ghana [29, 35], one in South Africa [36], two in Uganda [37, 38], and one in Tanzania [39]. Five studies were RCTs [29, 33, 34, 37, 38], three were evaluation and pre-post design studies [31, 32, 39], and two were mixed methods/cross-sectional studies [35, 36]. Seven studies occurred in a hospital setting [31–32, 36–38], one was school-based [29] and two were community-based [35, 39] (Table 1). Characteristics of studies All 5 RCTs were judged at low risk of bias for most criteria [29, 33, 34, 37, 38]. All reported random sequence generation except for one study [38], and all reported adequate method of concealment, losses to follow up and prespecified outcomes [29, 33, 34, 37, 38]. None of the RCTs reported blinding of participants. Almost all the five non-randomised studies showed a low risk of bias for selection of participants, measurement of interventions, departures from intended interventions. However, how confounding factors were controlled for and missing data were dealt with was not reported in any of the studies [31, 32, 35, 36, 39]. The measurement of outcomes in all the studies was based on a self-reported measure except for one study [37]. One of the non-randomised studies had significant technological challenges and inadequacies concerning the design of text message content, which could have influenced the outcome of the study [35]. Finally, only five studies out of ten analysed by intention-to-treat i.e. analysed the participants according to the groups to which they were originally assigned [29, 33, 34, 37, 38], which provides a more accurate, unbiased estimate of the findings reported in this study. mHealth interventions and platforms used. The most used mHealth platform was the Short Message Service (SMS) (9 studies). Only one of the studies used an interactive web-based peer support platform [31]. Five studies were based on unidirectional and interactive messaging services; three were based solely on 2-way interactive and two on unidirectional messaging services. The interventions focused on shaping knowledge and increasing the use of reproductive health interventions or services. Two studies evaluated SRH knowledge [29, 35], four assessed contraceptive use/birth control [29, 33, 34, 39], three examined pregnancy and fertility intentions [29, 33, 34]. One focused on facility childbirth delivery [33], two on exclusive breastfeeding (EBF) [33, 34], four on HIV Antiretroviral Therapy (ART) adherence [31, 32, 37, 38], and two on sexual behaviour [29, 34]. mHealth interventions and platforms used. The most used mHealth platform was the Short Message Service (SMS) (9 studies). Only one of the studies used an interactive web-based peer support platform [31]. Five studies were based on unidirectional and interactive messaging services; three were based solely on 2-way interactive and two on unidirectional messaging services. The interventions focused on shaping knowledge and increasing the use of reproductive health interventions or services. Two studies evaluated SRH knowledge [29, 35], four assessed contraceptive use/birth control [29, 33, 34, 39], three examined pregnancy and fertility intentions [29, 33, 34]. One focused on facility childbirth delivery [33], two on exclusive breastfeeding (EBF) [33, 34], four on HIV Antiretroviral Therapy (ART) adherence [31, 32, 37, 38], and two on sexual behaviour [29, 34]. Effectiveness of the mHealth interventions on improving SRH outcomes SRH knowledge. Table 2 shows the effect of the intervention on SRH knowledge, sexual behaviour, and contraceptive use. Three studies examined adolescent SRH knowledge outcomes [29, 31, 35]. Only one study evaluating a unidirectional and interactive intervention among adolescent girls showed a positive effect at 3-months and 15 months after controlling for covariates, (age, religion, ethnicity, parents’ educational level [29]. Sexual and Reproductive Health (SRH) knowledge increased by 11% (95% Confidence Interval (CI): 7%, 15%) and 24% (95% CI: 19, 28), greater than in the control group, respectively. This effect was maintained at 15 months in the interactive group, however, ceased to be significant among the unidirectional group. The remaining two studies showed a difference in knowledge between the intervention and control groups [31, 35]. However, the intervention group in one of the studies reported more false answers on SRH knowledge about STIs, abortion & contraception (1.7) compared to the control group (1.9) [35]. Similarly, with reference to improvement in HIV/ART knowledge, there was no statistically significant change in knowledge among adolescents who participated in the interactive web-based intervention (adolescents demonstrated less knowledge at end-line comparing to baseline) and those that did not take part [31]. Sexual health behaviour. Two studies reported effects on adolescent’s sexual behaviour [29, 34]. One study found that the SMS intervention influenced young women’s resumption of sexual intercourse such that most women (31.8% at 6 weeks, 57.9% at 14 weeks, and 67.7% at 6 months) reported having resumed sexual intercourse by 6 months [34]. Similarly, in another study, the interactive intervention was positively associated with having sex without a condom among sexually active adolescents in the interactive group (OR = 3.47; 95% CI = 1.12, 10.74). However, the intervention did not influence the age of sexual debut for those who have ever had sexual intercourse [29]. (See Table 2). Contraceptive/Birth control access and use. Four studies reported on contraceptive use [29, 33, 34, 39]. Evidence from the four studies showed that the intervention increased the use and access to contraceptive services and family planning initiation among adolescents and this was higher among the interactive compared with unidirectional group (Table 2). One study found that highly effective contraceptive (HEC) use at 6 months postpartum was significantly higher among those in the SMS group (69.9%) than in the control group (57.4%) ([RR] = 1.22; 95%; 1.01, 1.47; P = .04) [32]. Another study reported that contraceptive use was significantly higher in both intervention arms by 16 weeks (1-way SMS: 72% and 2-way SMS: 73%; p = 0·03 and 0·02 versus 57% control, respectively) [33]. This trend was reported in another study which found that the interactive intervention increased the odds of using oral contraceptives (OR = 13.2; 95% CI = 1.08, 161) and decreased the odds of using emergency contraception (OR = 0.22; 95% CI = 0.05, 0.88) [29]. Likewise, in another study, participants who engaged with the intervention accessed contraceptive information more frequently than non-intervention group [39]. Antiretroviral therapy (ART) adherence. Table 2 presents the results of the effects of the interventions on antiretroviral therapy adherence (ART), pregnancy & childbirth and breast feeding. Four studies reported effects on ART adherence [31, 32, 37, 38]. Overall, the four studies showed an improvement in adherence and pre-exposure prophylaxis (PrEP) initiation. However, this improvement was not statistically significant except for one study [32]. Despite relying on a routine collected measure, an evaluative study found that women who enrolled in the intervention were almost twice more likely to continue PrEP (22% vs. 43%; aRR = 1.75; 95% CI = 1.21, 2.55; P = .003), than women who initiated PrEP in the month before the intervention implementation [32]. This is contradicted by another study which showed no statistical difference in adherence between the intervention and control groups (Adherence was 64% for the 1-way group [OR = 0.64; 95% CI: 0.58, 0.70; P = .27] and 61% for the 2-way group [0.56, 0.67; P = .15], compared with 67% in the control group (OR = 0.67; 95% CI:0.62, 0.72] [35]. Also, there was no statistically difference between the proportion of participants achieving adherence of at least 90% over the 48-week period of analysis (1-way group = 28%; 2-way group = 26% and control = 29%; P = .85 and .69, respectively). A similar study found that at baseline, 71.6% of participants reported not to have missed any doses in the last week, while 77.8% of the participants at the post intervention reported not to have missed any doses in the last week [31]. Although not statistically significant (p = 0.95) and finally, this level of insignificance persists in another study, which found that after controlling for baseline adherence, the intervention group 1 (T1) had 3.8% lower adherence than the control group (95% CI -9.9, 2.3) and the Intervention group 2 (T2) had 2.4 percentage points higher adherence than the control group (95% CI -3.0, 7.9). However, the differences were not statistically significant for either intervention groups [38]. Pregnancy and childbirth. Three studies reported the effect of the intervention on pregnancy and childbirth outcomes [29, 33, 34]. These RCTs studies showed that the intervention influenced fertility intentions, reduced the odds of self-reported pregnancy and facility delivery. However, the effects were not statistically significant between the intervention and the control groups except for one study [29], where both the unidirectional and the interactive groups significantly lowered the odds of self-reported pregnancy by 86% in the adjusted models (odds ratio [OR] = 0.14; 95% CI = 0.03, 0.71) and 85% (OR = 0.15; 95% CI = 0.03, 0.86), respectively, compared with the control group. In another study of 184 participants who initially wanted to become pregnant again and whose fertility intentions were similar, found that after 6-month visits, fertility intentions were similar between groups with 26.2% who reported a desire to stop childbearing [34]. A similar study stated that at 10 weeks, facility delivery was high in all 3 intervention arms [33]. Among 277 women providing delivery data, 273 (98.6%) reported delivering in a facility, with no difference between the 1-way and control arms [relative risk (RR) 1.00, 95% CI 0·97–1·03; p = 0·99] or 2-way and control arms [RR 0·99, 95% CI 0·95–1·03; p = 0·54]. Although there were apparently fewer still-births and infant deaths in the 2-way group compared to the control group (3·1% versus 8%), but not statistically significant (p = 0.21) [31]. No serious adverse events occurred because of the intervention although one maternal death occurred (See Table 3). Breastfeeding. As reported in Table 3, only two RCTs reported the effects of SMS intervention on exclusive breastfeeding (EBF) [33, 34]. Both studies reported inconsistent findings with one study showing a significant improvement in EBF among the intervention group than the control group [33], and another showing no significant difference across the two groups examined [34]. One of the studies revealed that women in both intervention arms were significantly more likely to report EBF at 10 and 16 weeks than women in the control arm [10 weeks: Control arm (RR: 0.79; CL:0.69-.86); 1-way SMS RR = 0.93 (CI:0.86–0.97; p = 0.003), 2-way SMS: RR = 0.96 (0.89–0.98; P = 0.0004); At 16 weeks: Control arm [RR = 0.62; CI: 0.52–0.71]; 1-way SMS [RR = 0.82; CI 0.72–0.89, P = 0.002], 2-way SMS [RR = 0.93, CI: 0.85–0.97; P = 0.001]. At 24 weeks, the probability of EBF was higher in both intervention groups than in the control, but only statistically significant in the 2-way messaging group [0·49 in 1-way, 0·62 in 2-way, and 0·41 in control, (p = 0·30 and 0·005 for 1-way and 2-way vs. control, respectively). SRH knowledge. Table 2 shows the effect of the intervention on SRH knowledge, sexual behaviour, and contraceptive use. Three studies examined adolescent SRH knowledge outcomes [29, 31, 35]. Only one study evaluating a unidirectional and interactive intervention among adolescent girls showed a positive effect at 3-months and 15 months after controlling for covariates, (age, religion, ethnicity, parents’ educational level [29]. Sexual and Reproductive Health (SRH) knowledge increased by 11% (95% Confidence Interval (CI): 7%, 15%) and 24% (95% CI: 19, 28), greater than in the control group, respectively. This effect was maintained at 15 months in the interactive group, however, ceased to be significant among the unidirectional group. The remaining two studies showed a difference in knowledge between the intervention and control groups [31, 35]. However, the intervention group in one of the studies reported more false answers on SRH knowledge about STIs, abortion & contraception (1.7) compared to the control group (1.9) [35]. Similarly, with reference to improvement in HIV/ART knowledge, there was no statistically significant change in knowledge among adolescents who participated in the interactive web-based intervention (adolescents demonstrated less knowledge at end-line comparing to baseline) and those that did not take part [31]. Sexual health behaviour. Two studies reported effects on adolescent’s sexual behaviour [29, 34]. One study found that the SMS intervention influenced young women’s resumption of sexual intercourse such that most women (31.8% at 6 weeks, 57.9% at 14 weeks, and 67.7% at 6 months) reported having resumed sexual intercourse by 6 months [34]. Similarly, in another study, the interactive intervention was positively associated with having sex without a condom among sexually active adolescents in the interactive group (OR = 3.47; 95% CI = 1.12, 10.74). However, the intervention did not influence the age of sexual debut for those who have ever had sexual intercourse [29]. (See Table 2). Contraceptive/Birth control access and use. Four studies reported on contraceptive use [29, 33, 34, 39]. Evidence from the four studies showed that the intervention increased the use and access to contraceptive services and family planning initiation among adolescents and this was higher among the interactive compared with unidirectional group (Table 2). One study found that highly effective contraceptive (HEC) use at 6 months postpartum was significantly higher among those in the SMS group (69.9%) than in the control group (57.4%) ([RR] = 1.22; 95%; 1.01, 1.47; P = .04) [32]. Another study reported that contraceptive use was significantly higher in both intervention arms by 16 weeks (1-way SMS: 72% and 2-way SMS: 73%; p = 0·03 and 0·02 versus 57% control, respectively) [33]. This trend was reported in another study which found that the interactive intervention increased the odds of using oral contraceptives (OR = 13.2; 95% CI = 1.08, 161) and decreased the odds of using emergency contraception (OR = 0.22; 95% CI = 0.05, 0.88) [29]. Likewise, in another study, participants who engaged with the intervention accessed contraceptive information more frequently than non-intervention group [39]. Antiretroviral therapy (ART) adherence. Table 2 presents the results of the effects of the interventions on antiretroviral therapy adherence (ART), pregnancy & childbirth and breast feeding. Four studies reported effects on ART adherence [31, 32, 37, 38]. Overall, the four studies showed an improvement in adherence and pre-exposure prophylaxis (PrEP) initiation. However, this improvement was not statistically significant except for one study [32]. Despite relying on a routine collected measure, an evaluative study found that women who enrolled in the intervention were almost twice more likely to continue PrEP (22% vs. 43%; aRR = 1.75; 95% CI = 1.21, 2.55; P = .003), than women who initiated PrEP in the month before the intervention implementation [32]. This is contradicted by another study which showed no statistical difference in adherence between the intervention and control groups (Adherence was 64% for the 1-way group [OR = 0.64; 95% CI: 0.58, 0.70; P = .27] and 61% for the 2-way group [0.56, 0.67; P = .15], compared with 67% in the control group (OR = 0.67; 95% CI:0.62, 0.72] [35]. Also, there was no statistically difference between the proportion of participants achieving adherence of at least 90% over the 48-week period of analysis (1-way group = 28%; 2-way group = 26% and control = 29%; P = .85 and .69, respectively). A similar study found that at baseline, 71.6% of participants reported not to have missed any doses in the last week, while 77.8% of the participants at the post intervention reported not to have missed any doses in the last week [31]. Although not statistically significant (p = 0.95) and finally, this level of insignificance persists in another study, which found that after controlling for baseline adherence, the intervention group 1 (T1) had 3.8% lower adherence than the control group (95% CI -9.9, 2.3) and the Intervention group 2 (T2) had 2.4 percentage points higher adherence than the control group (95% CI -3.0, 7.9). However, the differences were not statistically significant for either intervention groups [38]. Pregnancy and childbirth. Three studies reported the effect of the intervention on pregnancy and childbirth outcomes [29, 33, 34]. These RCTs studies showed that the intervention influenced fertility intentions, reduced the odds of self-reported pregnancy and facility delivery. However, the effects were not statistically significant between the intervention and the control groups except for one study [29], where both the unidirectional and the interactive groups significantly lowered the odds of self-reported pregnancy by 86% in the adjusted models (odds ratio [OR] = 0.14; 95% CI = 0.03, 0.71) and 85% (OR = 0.15; 95% CI = 0.03, 0.86), respectively, compared with the control group. In another study of 184 participants who initially wanted to become pregnant again and whose fertility intentions were similar, found that after 6-month visits, fertility intentions were similar between groups with 26.2% who reported a desire to stop childbearing [34]. A similar study stated that at 10 weeks, facility delivery was high in all 3 intervention arms [33]. Among 277 women providing delivery data, 273 (98.6%) reported delivering in a facility, with no difference between the 1-way and control arms [relative risk (RR) 1.00, 95% CI 0·97–1·03; p = 0·99] or 2-way and control arms [RR 0·99, 95% CI 0·95–1·03; p = 0·54]. Although there were apparently fewer still-births and infant deaths in the 2-way group compared to the control group (3·1% versus 8%), but not statistically significant (p = 0.21) [31]. No serious adverse events occurred because of the intervention although one maternal death occurred (See Table 3). Breastfeeding. As reported in Table 3, only two RCTs reported the effects of SMS intervention on exclusive breastfeeding (EBF) [33, 34]. Both studies reported inconsistent findings with one study showing a significant improvement in EBF among the intervention group than the control group [33], and another showing no significant difference across the two groups examined [34]. One of the studies revealed that women in both intervention arms were significantly more likely to report EBF at 10 and 16 weeks than women in the control arm [10 weeks: Control arm (RR: 0.79; CL:0.69-.86); 1-way SMS RR = 0.93 (CI:0.86–0.97; p = 0.003), 2-way SMS: RR = 0.96 (0.89–0.98; P = 0.0004); At 16 weeks: Control arm [RR = 0.62; CI: 0.52–0.71]; 1-way SMS [RR = 0.82; CI 0.72–0.89, P = 0.002], 2-way SMS [RR = 0.93, CI: 0.85–0.97; P = 0.001]. At 24 weeks, the probability of EBF was higher in both intervention groups than in the control, but only statistically significant in the 2-way messaging group [0·49 in 1-way, 0·62 in 2-way, and 0·41 in control, (p = 0·30 and 0·005 for 1-way and 2-way vs. control, respectively). Components and characteristics of mHealth interventions Behavioural change components of the interventions. Overall, 23 from a possible list of 93 BCTs were identified as intervention components in the included studies (Fig 2). The 23 BCTs were from six out of the 16 possible domains (feedback & monitoring, social support, shaping knowledge, natural consequences reward & threat) of BCTs [23]. The most commonly used BCTs in these studies were feedback & monitoring, and social support (6 studies). The feedback and monitoring techniques mostly used in the studies focused on monitoring and providing informative feedback on scores and performance of the behaviour among participants. However, the feedback was based on change in knowledge and not on change in behaviour. Half of the studies (five out of ten) described how participants were socially and practically supported to achieve the intervention objectives, although some of the studies did not specify the nature of social support provided, and four studies reported on shaping knowledge through instruction on how to perform a behaviour. Two interventions did not report the use of any BCTs [36, 39]. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Heat map: Showing the behavioural techniques used as intervention components in each study. Key: 1 = Goals & planning, 2 = Feedback & monitoring, 3 = Social support, 4 = Shaping knowledge, 5 = Natural consequences, 6 = Comparison of behaviour, 7 = Associations 8 = Repetition & substitution, 9 = Comparison of outcomes, 10 = Reward & threat, 11 = Regulation, 12 = Antecedents, 13 = Identity, 14 = Scheduled consequences. 15 = Self-belief, 16 = Covert learning. https://doi.org/10.1371/journal.pone.0261973.g002 The second aspect was to identify the mHealth intervention content for the included studies: where it is being implemented (context), and how it was implemented (technical features) to support replication of the intervention using the mHealth evidence reporting and assessment (mERA) guidelines [24]. Fig 3 below shows the number of included studies meeting each mHealth criterion. On average, about 35% (6%-63%) of the 16 mERA criteria was achieved among all the 10 studies. Overall, most studies described the mode and frequency of intervention delivery [29, 31–39], how people were informed of the programme [29, 31–39], how the content of the intervention was developed [29, 31–34, 36–39] and the technology platform/software used in the programme implementation [29, 31–34, 37–39]. However, there were limited information on the barriers/challenges faced by participants in adopting the intervention [36] study), the physical infrastructure used to support the interventions [32, 34] and the security and confidentiality protocol of the interventions [33, 34]. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Number of papers that met mERA essential criteria among the 10 selected studies. No of studies in each component: Infrastructure [32, 42]; Technology platform [29, 31–34, 37–39]; Interoperability [29, 31–34, 38]; Intervention delivery [29, 31–39]; Intervention content [29, 31–34, 36–39]; Usability testing [29, 31–35, 37]; User feedback [31, 32, 38, 39]; Access of individual participants [38]; Cost assessment [29, 34]; Adoption inputs/programme entry [29, 31–39]; Limitations for delivery at scale [29, 35–38]; Contextual adaptability [31–35, 38]; Replicability [29, 32–34, 37, 38]; Data security [33, 34]; Compliance [29, 32, 34, 36, 38, 39]; Fidelity of the intervention [29, 32–34, 37–39]. https://doi.org/10.1371/journal.pone.0261973.g003 Behavioural change components of the interventions. Overall, 23 from a possible list of 93 BCTs were identified as intervention components in the included studies (Fig 2). The 23 BCTs were from six out of the 16 possible domains (feedback & monitoring, social support, shaping knowledge, natural consequences reward & threat) of BCTs [23]. The most commonly used BCTs in these studies were feedback & monitoring, and social support (6 studies). The feedback and monitoring techniques mostly used in the studies focused on monitoring and providing informative feedback on scores and performance of the behaviour among participants. However, the feedback was based on change in knowledge and not on change in behaviour. Half of the studies (five out of ten) described how participants were socially and practically supported to achieve the intervention objectives, although some of the studies did not specify the nature of social support provided, and four studies reported on shaping knowledge through instruction on how to perform a behaviour. Two interventions did not report the use of any BCTs [36, 39]. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Heat map: Showing the behavioural techniques used as intervention components in each study. Key: 1 = Goals & planning, 2 = Feedback & monitoring, 3 = Social support, 4 = Shaping knowledge, 5 = Natural consequences, 6 = Comparison of behaviour, 7 = Associations 8 = Repetition & substitution, 9 = Comparison of outcomes, 10 = Reward & threat, 11 = Regulation, 12 = Antecedents, 13 = Identity, 14 = Scheduled consequences. 15 = Self-belief, 16 = Covert learning. https://doi.org/10.1371/journal.pone.0261973.g002 The second aspect was to identify the mHealth intervention content for the included studies: where it is being implemented (context), and how it was implemented (technical features) to support replication of the intervention using the mHealth evidence reporting and assessment (mERA) guidelines [24]. Fig 3 below shows the number of included studies meeting each mHealth criterion. On average, about 35% (6%-63%) of the 16 mERA criteria was achieved among all the 10 studies. Overall, most studies described the mode and frequency of intervention delivery [29, 31–39], how people were informed of the programme [29, 31–39], how the content of the intervention was developed [29, 31–34, 36–39] and the technology platform/software used in the programme implementation [29, 31–34, 37–39]. However, there were limited information on the barriers/challenges faced by participants in adopting the intervention [36] study), the physical infrastructure used to support the interventions [32, 34] and the security and confidentiality protocol of the interventions [33, 34]. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Number of papers that met mERA essential criteria among the 10 selected studies. No of studies in each component: Infrastructure [32, 42]; Technology platform [29, 31–34, 37–39]; Interoperability [29, 31–34, 38]; Intervention delivery [29, 31–39]; Intervention content [29, 31–34, 36–39]; Usability testing [29, 31–35, 37]; User feedback [31, 32, 38, 39]; Access of individual participants [38]; Cost assessment [29, 34]; Adoption inputs/programme entry [29, 31–39]; Limitations for delivery at scale [29, 35–38]; Contextual adaptability [31–35, 38]; Replicability [29, 32–34, 37, 38]; Data security [33, 34]; Compliance [29, 32, 34, 36, 38, 39]; Fidelity of the intervention [29, 32–34, 37–39]. https://doi.org/10.1371/journal.pone.0261973.g003 Secondary outcomes Acceptability of mHealth interventions. Four studies evaluated adolescent acceptability of receiving an mHealth intervention for increasing adherence to HIV prevention or treatment [31, 32, 36, 38]. Three studies were among HIV positive adolescents [31, 36, 38] and one was pre-exposure prophylaxis among pregnant or post-partum women [32]. No study reported on the acceptability of mHealth interventions by parents of adolescents. Across all four of the studies, participants showed a positive attitude, and were willing to use and recommend mHealth interventions to others. One study reported that almost all the women (95%) would recommend the intervention (mWACh-PrEP) to other women who use pre-exposure prophylaxis (PrEP), and 95% would also use the program again if offered [32]. Also, 94% and 89% of women reported that SMS helped them understand and adhere to PrEP, respectively [32]. Another study found that among 81 adolescents with HIV that completed a mid-term questionnaire, the majority (95%) agreed to use the web-based interventions (ELIMIKA website) again and 87% would recommend it to others [31]. A related study in Uganda, showed that participants had positive attitudes about SMS as an incentive to adherence to ART (SITA) [38]. At follow-up, 96.6% of adolescents with HIV reported that they would remain in the intervention group if they had the choice (95.3% in the treatment 1 (T1) group and 97.8% in treatment 2 (T2) group, and 84.2% said there was nothing about SITA that they did not like (86.0% in T1 and 82.6% in T2). Participants from both intervention groups felt that SITA boosted their morale and prompted them to take their ART medication on time. Despite a small sample size, evidence from another study revealed that 65% of adolescents with HIV were willing to participate in a mHealth intervention to support treatment adherence [36]. Feasibility of delivery of mHealth interventions. Three studies reported on the feasibility of delivering mHealth intervention to adolescents in SSA [36, 38, 39]. All three of the studies established feasibility of delivering HIV treatment adherence and contraceptives information to adolescents using mHealth interventions. One study showed that sending text messages with information on a participant’s own adherence, information about the adherence performance of their peers and the recruitment process was practicable among HIV positive youth [38]. Also, a study found that use of text messages to support treatment adherence in adolescents with HIV was feasible, especially among in-school adolescents with high ownership of mobile phone with 67% willing to receive health related SMS [36]. Another study demonstrated that text messages comprising comprehensive information on contraceptive methods can be feasibly delivered and accessed by men and women of reproductive age [39]. However, Fig 3 (mERA checklist) above shows that only five studies reported the appropriateness of the intervention to the context and any possible adaptations required. Cost-effectiveness of mHealth interventions. None of the included studies reported cost-effectiveness outcomes. Intervention costs was among the least reported components in the mERA checklist (Fig 3). One study reported that the marginal costs of the interactive and unidirectional component per participant were US $1.91 and US $0.30, respectively [29]. Another study reported that the intervention was “relatively inexpensive “but with no information was provided on the specific costs of the intervention [34]. Acceptability of mHealth interventions. Four studies evaluated adolescent acceptability of receiving an mHealth intervention for increasing adherence to HIV prevention or treatment [31, 32, 36, 38]. Three studies were among HIV positive adolescents [31, 36, 38] and one was pre-exposure prophylaxis among pregnant or post-partum women [32]. No study reported on the acceptability of mHealth interventions by parents of adolescents. Across all four of the studies, participants showed a positive attitude, and were willing to use and recommend mHealth interventions to others. One study reported that almost all the women (95%) would recommend the intervention (mWACh-PrEP) to other women who use pre-exposure prophylaxis (PrEP), and 95% would also use the program again if offered [32]. Also, 94% and 89% of women reported that SMS helped them understand and adhere to PrEP, respectively [32]. Another study found that among 81 adolescents with HIV that completed a mid-term questionnaire, the majority (95%) agreed to use the web-based interventions (ELIMIKA website) again and 87% would recommend it to others [31]. A related study in Uganda, showed that participants had positive attitudes about SMS as an incentive to adherence to ART (SITA) [38]. At follow-up, 96.6% of adolescents with HIV reported that they would remain in the intervention group if they had the choice (95.3% in the treatment 1 (T1) group and 97.8% in treatment 2 (T2) group, and 84.2% said there was nothing about SITA that they did not like (86.0% in T1 and 82.6% in T2). Participants from both intervention groups felt that SITA boosted their morale and prompted them to take their ART medication on time. Despite a small sample size, evidence from another study revealed that 65% of adolescents with HIV were willing to participate in a mHealth intervention to support treatment adherence [36]. Feasibility of delivery of mHealth interventions. Three studies reported on the feasibility of delivering mHealth intervention to adolescents in SSA [36, 38, 39]. All three of the studies established feasibility of delivering HIV treatment adherence and contraceptives information to adolescents using mHealth interventions. One study showed that sending text messages with information on a participant’s own adherence, information about the adherence performance of their peers and the recruitment process was practicable among HIV positive youth [38]. Also, a study found that use of text messages to support treatment adherence in adolescents with HIV was feasible, especially among in-school adolescents with high ownership of mobile phone with 67% willing to receive health related SMS [36]. Another study demonstrated that text messages comprising comprehensive information on contraceptive methods can be feasibly delivered and accessed by men and women of reproductive age [39]. However, Fig 3 (mERA checklist) above shows that only five studies reported the appropriateness of the intervention to the context and any possible adaptations required. Cost-effectiveness of mHealth interventions. None of the included studies reported cost-effectiveness outcomes. Intervention costs was among the least reported components in the mERA checklist (Fig 3). One study reported that the marginal costs of the interactive and unidirectional component per participant were US $1.91 and US $0.30, respectively [29]. Another study reported that the intervention was “relatively inexpensive “but with no information was provided on the specific costs of the intervention [34]. Discussion Overall, the review demonstrates that mHealth interventions improve adolescent’s uptake of SRH services across a wide range of services. The evidence was strongest for increasing adolescent’s use of contraceptives. This is consistent with the results of previous reviews outside SSA [18, 19, 40]. For other SRH outcomes, the evidence was inconsistent. There was only one study that demonstrated a significant effect of mHealth interventions for each of the following outcomes: improving sexual health knowledge, adherence to HIV treatment, self-reported pregnancy, exclusive breastfeeding, delay of resumption of sexual activities for postpartum young women and increase in health facility delivery among adolescents, which is insufficient to establish the effectiveness of the interventions on these outcomes. Evidence from previous reviews conducted in high- and middle-income countries shows that mHealth interventions significantly improve SRH knowledge among adolescents [19]. Surprisingly, while there was an improvement on adolescent’s uptake of SRH services across a wide range of studies, one of the studies indicated increase in sex without having a condom among sexually active adolescents in the intervention group. The reason for this is quite unclear, and could be an artefact given that it was only one out of ten studies that reported negative effects of mHealth in this review. Most of the studies that had significant effects on improving uptake of SRH services among adolescents were those with two-way interactive components rather than one-way messaging services. Furthermore, interventions with more BCTs showed stronger efficacy than those with limited BCTs. This indicates that integration of effective BCTS and interactive components in future mHealth interventions may lead to more effective interventions [41]. The non-significant effects of some of the interventions in improving uptake of SRH services by adolescents in this study could arguably be attributable to the limited active ingredients of BCTs in these studies. Our results show that only 23 out of possible 93 BCTs were captured in the included studies and in some cases, there was no single element of BCTs in the intervention. The integration of active BCTs ingredients plays an important part in ensuring the interventions exert their effect [42] and bring about the desired change in the target behaviour [43]. This is because previous studies have shown that BCTs have been identified for interventions that prevent sexually transmitted infections (STIs) [44] and improve use of condom [45]. Our results showed that only few studies reported the challenges faced by participants in adopting the interventions, the physical infrastructure used to support the interventions and the security and confidentiality protocol of the interventions. This is concerning as information on these features could aid effective design of future mHealth intervention. The lower level of reporting completeness on essential features of mHealth interventions has been reported in previous reviews [19]. Although, the limited reporting could be attributable to the fact that the WHO developed mHealth reporting guideline is fairly new and insufficient reporting of mHealth features in studies published before the guideline was developed may be expected [19]. Finally, the results of our review showed that mHealth interventions to promote treatment adherence to prevent or treat HIV were acceptable to individuals, and can feasibly be delivered among adolescents in SSA. However, four of the five studies were non-randomised with moderate risk of bias. Despite the potential for digital interventions to be scalable and delivered at low cost, cost-effectiveness was not evaluated in any of the included studies. Furthermore, the cost implications of these mHealth interventions were among the least reported components of the mERA checklist. For the costs that were reported it was unclear if they referred to development costs, delivery costs or a combination of both. A similar review on the effectiveness of digital interventions on improving physical activity among adolescents also showed that none of the 32 included studies reported the cost effectiveness of the interventions [46]. Strengths and limitations Overall, the review followed an established guideline for undertaking reviews [25]. The literature search was comprehensive and identified a high number of potential studies including search of grey literature sources. The screening process was carried out by three independent reviewers, minimizing the risk of missing relevant studies. Data extraction and quality assessment were rigorous and transparent. Our findings were largely informed by high quality RCTs and non-randomised studies with low risk of bias. However, the measurement of outcomes in most of the studies was based on a self-reported measure, which could have introduced bias, which may have overestimated the treatment effect. It is important to note that few studies included older women in their analysis. For example, one of the papers [34] included women aged 14 and above in their intervention; making it difficult to disaggregate the data for young women aged 14–24 years. Also, caution should be exercised when interpreting the findings of the non-randomised studies given that most of them did not account for the missing data or controlling for confounding. Strengths and limitations Overall, the review followed an established guideline for undertaking reviews [25]. The literature search was comprehensive and identified a high number of potential studies including search of grey literature sources. The screening process was carried out by three independent reviewers, minimizing the risk of missing relevant studies. Data extraction and quality assessment were rigorous and transparent. Our findings were largely informed by high quality RCTs and non-randomised studies with low risk of bias. However, the measurement of outcomes in most of the studies was based on a self-reported measure, which could have introduced bias, which may have overestimated the treatment effect. It is important to note that few studies included older women in their analysis. For example, one of the papers [34] included women aged 14 and above in their intervention; making it difficult to disaggregate the data for young women aged 14–24 years. Also, caution should be exercised when interpreting the findings of the non-randomised studies given that most of them did not account for the missing data or controlling for confounding. Conclusions This review demonstrates that interactive mobile health interventions with effective behaviour change techniques have strong potential to improve adolescent uptake of health services. This evidence heightens the need to develop mHealth interventions tailored for adolescents, which are theoretically informed and incorporate effective behaviour change techniques. Such interventions could improve the use of sexual and reproductive health services and lead to health improvement among adolescents in SSA. Also, future research should prioritise transparent reporting of the essential components of mHealth interventions to support accurate generalisation, application of the findings, and replication of the intervention. Studies evaluating cost-effectiveness of mhealth interventions are required. Supporting information S1 Checklist. https://doi.org/10.1371/journal.pone.0261973.s001 (DOCX) S1 File. https://doi.org/10.1371/journal.pone.0261973.s002 (DOCX)

Mayne, Elizabeth S.;Mayne, Anthony;Louw, Susan

doi: 10.1371/journal.pone.0262306pmid: 35061794

Introduction Disseminated intravascular Coagulation (DIC) is a thrombotic microangiopathy which may complicate a number of severe disease processes including sepsis. Development of microvascular thromboses results in consumption of coagulation factors and platelets and ultimate bleeding. Patients with HIV infection (PWH) often present with baseline dysregulation of the coagulation system which may increase severity and derangement of DIC presentation. Previously, we have shown that HIV is a significant risk factor for development of DIC. Methodology We conducted a retrospective record review of all DIC screens submitted to our tertiary coagulation laboratory in Johannesburg, South Africa, over a one year period and compared the laboratory presentation of DIC in PWH with presentation of DIC in patients without HIV infection. Results Over the year, 246 patients fulfilled the International Society of Thrombosis and Haemostasis (ISTH) diagnostic criteria for DIC– 108 were confirmed HIV-infected and 77 were confirmed uninfected. PWH and DIC presented at a significantly earlier age (41 vs 46 years respectively, p<0.02). The prothrombin time was significantly more prolonged (30.1s vs 26.s), the d-dimer levels were substantially higher (5.89mg/L vs 4.52mg/L) and the fibrinogen (3.92g/L vs 1.73g/L) and platelet levels (64.8 vs 114.8x109/l) were significantly lower in PWH. PWH also showed significant synthetic liver dysfunction and higher background inflammation. Conclusion PWH who fulfil the diagnostic criteria for DIC show significantly more dysregulation of the haemostatic system. This may reflect baseline abnormalities including endothelial dysfunction in the context of inflammation and liver dysfunction. Introduction South Africa has a high prevalence of Human immunodeficiency virus (HIV) infections with an estimated 13% or 7.8 million people with HIV (PWH) and approximately 110 000 HIV-related deaths annually [1]. The prevalence of HIV infection is even higher in the South African in-hospital population particularly in medical wards and intensive care units where the true prevalence may be over 60% [2, 3]. HIV infection status should be considered in all patients particularly since HIV can result in unusual presentations and exacerbations of pre-existing conditions [4]. HIV infection is strongly associated with pro-thrombotic states including pulmonary thromboembolic disease and deep vein thrombosis [5], arterial thrombosis manifesting as cardiovascular disease [6] and microvascular disorders including, most prominently, thrombotic thrombocytopaenic purpura (TTP) [7]. This increased propensity to pathological clotting has been attributed to a number of different features including endothelial dysfunction resulting from chronic inflammation (reviewed in [8]), associated infections (opportunistic and non-opportunistic), an imbalance between pro- and anti-coagulant factor levels [9–15] as well as platelet dysfunction [16]. These derangements, documented in individuals on antiretroviral therapy (ART) as well as in patients with uncontrolled viraemia (ART-naïve), include decreased Protein S levels, elevated levels of coagulation factors including factor VIII and von Willebrand factor [9, 14] and quantitative and qualitative platelet disorders [13, 16, 17]. The imbalance between pro- and anticoagulant factors manifests in the laboratory as elevated D-dimers levels in HIV infected individuals [18, 19]. Disseminated intravascular coagulation (DIC) is a thrombotic microangiopathic state characterised by widespread microvascular thrombosis, consumption of coagulation factors and platelets and ultimately a bleeding diathesis [20]. It is important to make the diagnosis of DIC in an appropriate clinical setting and a number of triggers have been associated with DIC development including severe infection and sepsis, malignancy, trauma and obstetric complications [21]. No single laboratory test is sufficiently robust, specific or sensitive enough to diagnose DIC. The diagnosis may be made using a number of scoring systems which assign numeric values to abnormalities in a panel of tests including the International Society of Thrombosis and Haemostasis (ISTH) Diagnostic Scoring system which assigns points for reduction in platelet count, elevation of fibrin degradation products, prolongation of the prothrombin time and the fibrinogen level (Table 1) [20]. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. The ISTH scoring system for overt DIC. https://doi.org/10.1371/journal.pone.0262306.t001 Although HIV is associated with the development of DIC, this has been considered an indirect relationship with the majority of HIV-associated DIC cases attributed to HIV- related infections or malignancies rather than with HIV as a primary pathophysiological trigger for DIC development. We have reported the high risk of development of DIC in HIV-infected patients even in the absence of other comorbidities [21]. It is important to define the trigger accurately as the primary therapy of DIC is the treatment of the underlying cause although supportive replacement of coagulation factors and platelets may also be considered especially if a patient is actively bleeding [22]. PWH often show baseline activation of the coagulation and haemostatic systems which may impact the laboratory presentation of these patients. In order to define the laboratory presentation of DIC in patients with and without HIV infection, we undertook a retrospective analysis of all DIC screens submitted to our tertiary care academic facility in Johannesburg, South Africa. Methodology Permission for this retrospective study was granted by the University of the Witwatersrand Human Research Ethics Committee (Certificate number: M160389). All data were retrieved from the laboratory information system for screens submitted to the coagulation laboratory at the National Health Laboratory Service (NHLS) Charlotte Maxeke Johannesburg Academic Hospital (CMJAH) for a one year period from 2015 to 2016. This laboratory is a specialist referral facility for the academic hospital and a number of surrounding hospitals in the greater Gauteng area. Since the study was a retrospective record review, no patient clinical data were obtained and no further testing was possible. All data were anonymized. The minimum dataset used included D-dimer levels, prothrombin time (PT), activated partial thromboplastin time (aPTT), platelet count and fibrinogen levels. The coagulation analysis was performed on a STAGO STA-R Max™ (Diagnostica Stago, Asnières-sur-Seine, France) and platelet counts were performed on the Sysmex XN analyser (Sysmex, Kobe, Japan). For each participant, the ISTH DIC scoring system was used to assess the presence or absence of DIC with patients with ISTH scores of 5 and greater considered to have overt DIC. Other data collected from the database included albumin levels (a surrogate of liver dysfunction), C-reactive protein levels, HIV status (where this had been performed) and CD4+ T cell counts and viral load levels in patients who were HIV infected. Screens where no testing for HIV had been conducted were excluded from the final analysis. Statistical analysis was performed using StataSE® 14.2 StataCorp. Summary statistics of all analytes including mean values and standard deviations were computed. Mean values were compared using a student’s t-test after applying ranking by an upward sort to analytes by positive or negative HIV status. A p-value <0.05 was considered significant. Results For the period, 246 patients met the ISTH diagnostic criteria for a DIC. Of these, 61 had no recorded HIV test and were excluded from further analysis. 108 patients were HIV-infected and 77 patients were confirmed HIV-uninfected. Of the 108 HIV-infected patients, 67 viral loads and 96 CD4+ T cell counts were available. It was not possible to assess treatment status or length of infection in these patients. The summary data are presented in Table 2. Patients with HIV-associated DIC presented at a significantly earlier age. The coagulation parameters were significantly more deranged although this did not impact the mean ISTH score. Both HIV-infected and uninfected patients presented with a high C-reactive protein, reflecting the high level of inflammation in both cohorts. Importantly, HIV-infected patients had significantly lower mean albumin levels. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Demographic and laboratory parameters in patients presenting with a DIC. https://doi.org/10.1371/journal.pone.0262306.t002 Discussion HIV-infected patients in South Africa often present for treatment with advanced disease at lower CD4+ T cell counts and with higher viral loads [23, 24]. This is associated both with an increased incidence of opportunistic and non-communicable complications of HIV infection and with longer and more severe chronic inflammation. This results in an underlying derangement of the coagulation pathways. Previously we have shown that HIV-infection can be a significant trigger for the development of the thrombotic microangiopathy (TMA), DIC [21]. In this study, we show that HIV-infected patients with DIC present at a younger age and with a more significant coagulation disorder. Although this did not significantly impact their DIC score, it is possible that the diagnosis of overt DIC may be made erroneously in these patients with significant baseline activation of the haemostatic systems. The implications of the severity of the coagulopathy on outcomes and management of these patients should be urgently investigated. Importantly, HIV-infected patients showed significantly lower mean albumin levels. Albumin is a negative acute phase reactant and is often reduced in severe infection. It may also, however, reflect liver synthetic dysfunction [25]. The majority of coagulation factors are synthesised in the liver and the prothrombin time, in particular, is a sensitive indicator of liver disease. Unfortunately, limited clinical data were available but liver involvement in HIV could be associated with opportunistic infections especially disseminated mycobacterial disease or malignancies like, for example, B-cell lymphoma [26–28]. As these conditions may also predispose patients to the development of DIC, this may contribute to diagnostic uncertainty in patients with HIV-associated DIC. This study has a number of limitations. There were limited clinical data and the outcomes of the patients included is unknown. In the majority of cases, serial measurements were not available and HIV viral loads and CD4+ T cell counts were also not available on all patients. This study does, however, indicate that DIC in PWH presents with more significant derangement of coagulation parameters and this should be considered when making the diagnosis of DIC in this population. Supporting information S1 File. Raw data for all quantitative DIC parameters. https://doi.org/10.1371/journal.pone.0262306.s001 (XLSX)

Guo, Qi;Sun, Yuhan;Zhang, Jiangtao;Li, Yun

doi: 10.1371/journal.pone.0262278pmid: 34986177

Introduction A germplasm resources is a collection of all the genes of a species’ individuals and play an important role in developing new varieties, discovering important agronomic traits, conserving endangered species, maintaining ecological balance and stabilizing the environment [1–3]. The germplasm can be considered as a carrier of biodiversity and is highly important [4]. In situ preservation and ex situ preservation are the two standard methods for the protection of germplasm resources. Although in situ conservation maintains the original ecosystem and natural habitat of plants better than ex situ preservation, it requires a large cultivation area and large investments of labor, materials, finances, and time for administration and management. Ex situ preservation acts as a backup for certain aspects of diversity that might otherwise be lost in human-dominant ecosystems and in nature [5–7]. Forest trees require a long period and large area for growth, and ex situ preservation is commonly used to protect plant resources. This method is convenient for breeders as it, allows research to be carried out in in a timely and efficient manner [8, 9]. Since the early 1990s, China began to carry out systematic research work on protecting forest genetic resources. A preservation system for forest genetic resources was established for country’s actual situation, by forming a framework for preserving and utilizing forest germplasm resources. It was coordinated by the National Forest Germplasm Resources Platform and National Forest Base. Breeding efforts have been carried out on many endangered species, such as Taxus chinensis var. Mairei [10, 11], dove tree (Davidia involucrata Baill) [12], Emmenopterys henryi Oliv. [13, 14], and Cathaya argyrophylla [15], so that populations of endangered species can expand. Forest germplasm resources with excellent characteristics and important economic value have been examined and approved for improved, new and local varieties. By this means, excellent seedlings and propagation materials should be produced by establishing seed standards, seed orchards, cutting orchards, demonstration forests, and so on. The goal of tree genetic resource preservation is to create ecological and social benefits to maintain the sustainable development of a biological environment. Nevertheless, the utilization of forest resources is still occurring at a slow pace, given the destruction of the environment and the demand for economic development. This situation can be reversed by paying attention to the investigation, protection and utilization of the forest germplasm; increasing funding support for forest resource research, improving the publicity and public education about forest genetic resources; and raising the whole population’s awareness to protect genetic resources [16–20]. Black locust (Robinia pseudoacacia L.) is a multipurpose deciduous tree species. It is suitable for land reclamation, windbreaks, fence posts, raw material for energy plantations, timber, bee-keeping, wood fibre, and forage [21, 22]. It was first introduced to China in 1877, and been extensively planted in 27 provinces [23, 24]. Robinia pseudoacacia L. has become an important pioneer afforestation tree in northwest China because of it can withstand slight saline-alkaline and dry soils and has grown well in barren mountains [25]. In addition, the black locust is an economically valuable tree: bees feed on its flowers to produce honey, the forest rapidly growths, and the durability and strength its timbers make it suitable for building [26–28]. However, the haphazard introduction of black locust into many areas of China and a related lack of records about introduced samples have led to confusion concerning the black locust germplasm resources in China. This has greatly restricted breeding efforts and effective utilization of the black locust. The germplasm resources of black locust were systematically utilized and protected until the target-oriented breeding stage began in the 1990s [29–32]. A prerequisite for plant breeding is the research of genetic variation among individuals and groups of individuals. Plant phenotypes and physiological traits are affected by environmental conditions. Therefore, different plant phenotypes and physiological indicators can reflect the degree of plant adaptability to current site conditions [33, 34]. Knowledge about trait variation can provide a more comprehensive understanding of germplasm resource diversity among breeding materials. It can be used to carry out further targeted breeding work based on the characteristics of each germplasm resource. The degree of variation and pattern of germplasm resources can be determined relatively easily. This, overall concept forms the basis of genetic breeding and is the most common and effective method for breeders [35–37]. Unfortunately, there have been few studies of the differences in the provenance of leaf phenotypic and physiological parameters of various black locust collected within their natural distribution [38]. However, an analysis based on morphological characteristics including germination ability, plant height and diameter at the ground level was performed to evaluated the seed characteristics and variation of 19 black locust from provenances in China. The results showed that interactions between the genotype and environment caused significant differences in tree growth from those provenances under different site conditions [39]. Zhang et al. [40] analyzed the seed survival rate, average height and diameter at breast height (DBH) of black locust trees from different Chinese provenances. They found that the further the provenance was from the test site, the lower its the survival and growth rates. At the origin provenance level, Zhou [41] investigated the differences in fruits, seeds, seedling height and diameter at the ground level for annual seedlings from different original black locust provenances. He also assessed the geographical variation in traits of different provenances, and made a preliminary selection of two provenances with high growth. Li et al. [42] surveyed 183 black locust families from 38 provenances and found that the regular geographic variation reflected in the origin of black locust’s cold resistance presented a meridional gradient, which increased with increasing longitude. To explore the diversity of leaf phenotypes and physiology based on natural black locust areas, 20 black locust provenances, 19 provenances covering almost the entire natural distribution and one provenance from China (CN), were used to comparatively analyze the variation among sixteen measured traits. Following that some elite trees were selected. The results provide valuable resources for efficient breeding and germplasm preservation of black locust trees in the future. Materials and methods Plant materials In this study, 214 samples of black locust were collected from 20 locations from September to October 2010. They comprised the 19 main black locust natural distribution areas in the United States and one main cultivation area in Henan, China (Table 1) [43]. In these areas, several fruits of black locusts with normal growth and a diameter at breast height (DBH) greater than 20 cm were collected at 500 m intervals and mailed to the Henan Academy of Forestry Sciences in China. From April to July 2011, these collected seeds were placed in a greenhouse for 24 h until germination. The successfully germinated seedlings were placed in nutrient bowls filled with nutritive soil and managed normally until the seedlings’ height reached approximately 30 cm high, and were then transplanted to Mengjin Forest Farm (34°49′18″N, 112°28′12″E), Luoyang, Henan, China. Mengjin County is a transition zone between subtropical and temperate; the annual average temperature was 15.4°C, and the annual average precipitation was 593 mm from 2010 to 2017, respectively. The soil is mainly brown soil (accounting for 93%), followed by alluvial soil (accounting for 7%). There was at least one site for each provenance, and each site contained at least two accessions randomly distributed with a plant spacing of 4 m×4 m. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Information on the 20 R. pseudoacacia L. provenances. https://doi.org/10.1371/journal.pone.0262278.t001 Two hundred and fourteen well-grown black locusts of different provenances were selected as experimental material from the Mengjin Forest Farm in August 2017. Each specimen was chosen by selecting those whose annual branches were free of pests and disease at the same height in four directions (north, south, east, and west). Data collection In this study, 16 quantitative traits of black locust trees were evaluated. They included 13 leaf traits: compound leaf length (CLL), compound leaf width (CLW), compound leaf length/width (CLL/CLW), compound petiole length (CPL), leaflet length (LL), leaflet width (LW), leaflet length/width (LL/LW), leaflet area (LA), leaflet perimeter (LPM), leaflet circularity (LC), leaflet pairs (LP), leaflet number (LN), and petiole angle (PA). The 16 traits also included three physiological traits: chlorophyll content (Chl), total protein content (Spro), and proline (PRO). The 13 leaf traits and Chl were evaluated at maturity in August 2017. The two remaining physiological traits were measured in healthy leaves from April to May 2017 (Table 2). The leaves were collected from trees and rapidly placed in sealed bags containing dry ice, and then transferred to the National Engineering Laboratory for Tree Breeding, Beijing Forestry University, China (40°0′22″N, 116°21′1″E), and stored at -80°C until tested. The compound leaf traits were measured by a ruler with a precision of 0.01 cm. The petiole angle was surveyed by an electronic protractor with a precision of 0.01°. The leaves were scanned and saved in the same manner, and the remaining leaf characteristics were analyzed using LAMINA version 1.0.2 software. The chlorophyll content in mature leaves of black locust was determined by a SPAD-502 Plus chlorophyll meter (Konica Minolta, Japan); three leaflets were selected randomly from each direction (north, south, east, and west), and each leaflet was measured at 6 positions. Following that, the results were averaged into one measurement. Total protein content and proline were evaluated with a Total Protein Assay Kit (Art. No. A045-4) and a Proline Assay Kit (Art. No. A107), produced by the Nanjing Jiancheng Bioengineering Institute (http://www.njjcbio.com/). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Investigated information of R. pseudoacacia L. germplasm. https://doi.org/10.1371/journal.pone.0262278.t002 Data and statistics Microsoft Excel 2016 was used to examine the variation in leaf phenotypic and physiological traits, including the mean value, standard error, amplitude, and coefficient of variation (CV). SPSS version 24 was used to perform analyses of variance (ANOVAs) in conjunction with Duncan’s multiple range tests for multiple comparisons. Principal component analysis was applied to the sixteen traits of the black locust provenances. A p-value for the ANOVA F tests ≤0.05 was considered significant. The formula, Vst (%) = δ2t/s/(δ2t/s+δ2s), was used to calculate the percentage of variance among and within the provenances, where Vst is the differentiation coefficient of the trait, δ2t/s is the variance component between provenances, and δ2s is the variance component within provenances [44]. A covariance correlation matrix was then used to analyze the correlations between clonal populations and geographical populations. The euclidean distance of each quantitative trait was calculated with the open-source statistical package, R; graphical visualization of the results was carried out using MEGA ver. 6.0 [45] after all the tested data had been processed in SPSS version 24. In addition, Mantel’s correlation tests were conducted on the euclidean and geographical distances of all the traits of the black locust trees. Plant materials In this study, 214 samples of black locust were collected from 20 locations from September to October 2010. They comprised the 19 main black locust natural distribution areas in the United States and one main cultivation area in Henan, China (Table 1) [43]. In these areas, several fruits of black locusts with normal growth and a diameter at breast height (DBH) greater than 20 cm were collected at 500 m intervals and mailed to the Henan Academy of Forestry Sciences in China. From April to July 2011, these collected seeds were placed in a greenhouse for 24 h until germination. The successfully germinated seedlings were placed in nutrient bowls filled with nutritive soil and managed normally until the seedlings’ height reached approximately 30 cm high, and were then transplanted to Mengjin Forest Farm (34°49′18″N, 112°28′12″E), Luoyang, Henan, China. Mengjin County is a transition zone between subtropical and temperate; the annual average temperature was 15.4°C, and the annual average precipitation was 593 mm from 2010 to 2017, respectively. The soil is mainly brown soil (accounting for 93%), followed by alluvial soil (accounting for 7%). There was at least one site for each provenance, and each site contained at least two accessions randomly distributed with a plant spacing of 4 m×4 m. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. Information on the 20 R. pseudoacacia L. provenances. https://doi.org/10.1371/journal.pone.0262278.t001 Two hundred and fourteen well-grown black locusts of different provenances were selected as experimental material from the Mengjin Forest Farm in August 2017. Each specimen was chosen by selecting those whose annual branches were free of pests and disease at the same height in four directions (north, south, east, and west). Data collection In this study, 16 quantitative traits of black locust trees were evaluated. They included 13 leaf traits: compound leaf length (CLL), compound leaf width (CLW), compound leaf length/width (CLL/CLW), compound petiole length (CPL), leaflet length (LL), leaflet width (LW), leaflet length/width (LL/LW), leaflet area (LA), leaflet perimeter (LPM), leaflet circularity (LC), leaflet pairs (LP), leaflet number (LN), and petiole angle (PA). The 16 traits also included three physiological traits: chlorophyll content (Chl), total protein content (Spro), and proline (PRO). The 13 leaf traits and Chl were evaluated at maturity in August 2017. The two remaining physiological traits were measured in healthy leaves from April to May 2017 (Table 2). The leaves were collected from trees and rapidly placed in sealed bags containing dry ice, and then transferred to the National Engineering Laboratory for Tree Breeding, Beijing Forestry University, China (40°0′22″N, 116°21′1″E), and stored at -80°C until tested. The compound leaf traits were measured by a ruler with a precision of 0.01 cm. The petiole angle was surveyed by an electronic protractor with a precision of 0.01°. The leaves were scanned and saved in the same manner, and the remaining leaf characteristics were analyzed using LAMINA version 1.0.2 software. The chlorophyll content in mature leaves of black locust was determined by a SPAD-502 Plus chlorophyll meter (Konica Minolta, Japan); three leaflets were selected randomly from each direction (north, south, east, and west), and each leaflet was measured at 6 positions. Following that, the results were averaged into one measurement. Total protein content and proline were evaluated with a Total Protein Assay Kit (Art. No. A045-4) and a Proline Assay Kit (Art. No. A107), produced by the Nanjing Jiancheng Bioengineering Institute (http://www.njjcbio.com/). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Investigated information of R. pseudoacacia L. germplasm. https://doi.org/10.1371/journal.pone.0262278.t002 Data and statistics Microsoft Excel 2016 was used to examine the variation in leaf phenotypic and physiological traits, including the mean value, standard error, amplitude, and coefficient of variation (CV). SPSS version 24 was used to perform analyses of variance (ANOVAs) in conjunction with Duncan’s multiple range tests for multiple comparisons. Principal component analysis was applied to the sixteen traits of the black locust provenances. A p-value for the ANOVA F tests ≤0.05 was considered significant. The formula, Vst (%) = δ2t/s/(δ2t/s+δ2s), was used to calculate the percentage of variance among and within the provenances, where Vst is the differentiation coefficient of the trait, δ2t/s is the variance component between provenances, and δ2s is the variance component within provenances [44]. A covariance correlation matrix was then used to analyze the correlations between clonal populations and geographical populations. The euclidean distance of each quantitative trait was calculated with the open-source statistical package, R; graphical visualization of the results was carried out using MEGA ver. 6.0 [45] after all the tested data had been processed in SPSS version 24. In addition, Mantel’s correlation tests were conducted on the euclidean and geographical distances of all the traits of the black locust trees. Results Analysis of leaf phenotypic characteristics and physiological indicators Table 3 shows the mean values and the multiple comparison results based on Duncan’s test. Compound leaves and leaflets differed between the 20 provenances. Between the four compound leaf traits, there was no significant difference in compound leaf length (CLL), and the length ranged from 26.426 cm (CN) to 29.710 cm (IL). Compared with those at the other locations, the accessions in MS, with the largest value of 2.916 cm, showed a significant difference in CLL/CLW traits. Of the nine leaflet characteristics, there were no significant differences in petiole angle (PA) between accessions in LA. The accessions in MS had the smallest LL and LP, whereas those in KS had the largest LP and LN, with averages of 9.971 and 20.833, respectively. Similarly, the results for the multiple comparison tests of the three physiological traits showed no significant difference in proline content between provenances, but both the chlorophyll content and soluble protein content had significant differences. Among the above indicators, the provenances in IN, KS, and OH showed the highest values of certain characteristics—38.22±0.69 (Chl-SPAD value), 1445.861±899.893 μg·ml-1 (Spro) and 44.649±11.401 μg·g-1 (PRO), respectively. However, the provenances in KS, AR, and OK had the lowest values—31.98±0.54 (Chl-SPAD value), 1445.861±899.893 μg·ml-1 (Spro), and 19.257±3.532 μg·g-1 (PRO), respectively. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Differences in leaf phenotypic traits of 20 R. pseudoacacia L. provenances. https://doi.org/10.1371/journal.pone.0262278.t003 Differentiation coefficient analysis of sixteen trait parameters The analysis results (S1 Table in S1 File, Fig 1) showed that the coefficient of variation in the 13 leaf phenotypic traits among the different provenances was 3.741%-19.599%. All traits were lowest in LC and highest in LA, which indicates that LC has the smallest dispersion degree and highest stability among the leaf phenotypic traits; the case of LA is completely opposite. The coefficient of variation of the compound leaf traits was 11.236% and close to that of the leaflet traits (11.301%). At the provenance level, the total coefficient of variation was 11.281%, with the lowest and highest coefficients of variation at 7.286% (MS/AL) and 14.788% (TN), respectively, which shows that the provenance in TN had the most abundant leaf phenotypic diversity among the 20 black locust provenances. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Distribution patterns of coefficients of variation of 16 tested traits of R. pseudoacacia L. in 20 provenances. https://doi.org/10.1371/journal.pone.0262278.g001 Similarly, the total average variation of the three physiological parameters was 30.993%, and the value of soluble proteins (Spro) (42.363%) had the highest level, followed by the proline (PRO) (42.356%) and chlorophyll content (Chl) (8.260%); Spro had the highest stability of these parameters, as revealed by the maximum-to-minimum ratios of each indicator—6.364%, 8.735% and 19.178%, respectively. Of the 20 provenances, KY provenance had the highest total average coefficient of variation (52.786%), which was approximately 3.8 times that for the MS provenance, which presented the smallest variation coefficient. In conclusion, the variation richness of the 16 test indexes of the 214 black locust accessions was as follows: Spro (42.363%)>PRO (42.356%)>LA (19.599%)>PA (17.317%)>CLW (12.359%)>CPL (11.179%)>LL (10.942%)>LPM (10.741%)>CLL (10.708%)>CLL/CLW (10.697%)>LP (10.558%)>LW (10.522%)>LN (9.832%)>LL/LW (8.462%)>Chl (8.260%)>LC (3.741%). Similarly, the variation in richness at each provenance level was as follows: KY (18.514%)>WV (18.411%)>VA (17.323%)>OH (17.222%)>TN (17.105%)>CN (16.231%)>PA (16.174%)>IN (15.317%) = IL (15.317%)>GA (15.275%)>KS (14.943%)>MS (14.632%)>IA (14.606%)>MO (14.098%)>MD (13.852%)>OK (13.304%)>AL (13.298%)>AR (13.127%)>MS/AL (10.571%)>NC (10.224%). Analysis of the differentiation coefficients of sixteen trait parameters The differentiation coefficient and variance components among/within provenances ranged from 25.843% (CLW) to 71.655% (PA) at the leaf phenotypic trait level, with an average of 48.829%. The mean leaf phenotypic differentiation coefficient of the nine leaflet traits was 52.259%, lower than that of four compound leaf indicators (41.112%). Thus the differentiation level for the leaflets was slightly higher than for the compound leaves. In terms of single traits, the differentiation coefficients of CPL, LW, LP, LN and PA were larger than those within provenances, indicating that the variation among provenances was higher than within them (Table 4). Furthermore, the total mean leaf phenotypic differentiation coefficient among provenances was lower than within provenances, suggesting that the main variation of black locust occurred intra-provenance, not inter-provenance variation. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 4. Variance percentages and differentiation coefficients of 16 traits among/within provenances of R. pseudoacacia L. https://doi.org/10.1371/journal.pone.0262278.t004 Homoplastically, the variation of the three physiological parameters showed that Spro had the lowest and PRO, the highest differentiation coefficient at—23.956% and 49.137%, respectively. The total mean differentiation coefficient of the three indicators was 40.383%, but it was 59.617% within provenances. These results are consistent with the findings of leaf phenotypic traits, further demonstrating the importance of individual variation in black locust trees. Correlation analysis of sixteen trait parameters Sixty-one (50.833%) significantly correlated traits were identified by Pearson correlation analyses of the 16 parameters (CLL, CLW, CLL/CLW, CPL, LL, LW, LL/LW, LA, LPM, LC, LP, LN, PA, Chl, Spro, and PRO) in 20 R. pseudoacacia provenances (P<0.05 or P<0.01) (S2 Table in S1 File, Fig 2). Among these traits, there were 50 significantly correlated leaf phenotypic traits (P<0.05). For compound leaf traits, CLL/CLW showed a highly significant negative correlation with CLW (P<0.01) but was not significantly correlated with CPL (P>0.05). In terms of leaflets, LC showed significant negative correlations with LW, LL/LW, and LPM, and LW was significantly positively correlated with LA, LPM, and LC (P<0.01). However, CLL/CLW showed significant negative correlations with LL, LW, LA, LPM, and LP (P<0.01). There were 10 significantly correlations between phenotypic and physiological traits. CPL and LW exhibited extremely significant negative correlations between all pairs of physiological traits; Chl presented significant positive correlations with LP and LN, and PRO was significantly negatively correlated with CLW and LA (P<0.05). There were no significant correlations between physiological traits, except for a correlation between Spro and PRO. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Heat map of a correlation analysis of 16 parameters from 214 R. pseudoacacia L. samples. Notes: *, P<0.05; **, P<0.01. https://doi.org/10.1371/journal.pone.0262278.g002 Principal component analysis of sixteen trait parameters Five principal components explained the investigated characteristics with eigenvalues greater than 1.0 (Table 5), and the cumulative contribution rate reached 80.004%, essentially reflecting the main information contained in all the measured indexes. In principal component 1, the CLL, CLW, CPL, LL, LW, LA, and LPM with relatively high absolute values and were representative of some compound leaf and leaflet characteristics; in principal component 2, LP and LN were relative; CLL/CLW, LL/LW and LC constituted the principal component 3; and the principal component 4 emphasized PA, Spro, and PRO. The fifth component separated Chl. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 5. Principal component analysis of 16 traits of different R. pseudoacacia L. geographic provenances. https://doi.org/10.1371/journal.pone.0262278.t005 Cluster analysis based on sixteen trait parameters Cluster analysis was performed for twenty different R. pseudoacacia provenances using the method of hierarchical cluster analysis between groups, taking sixteen trait parameters as variables (Fig 3). When the euclidean distance was set to 10, the twenty R. pseudoacacia provenances could be divided into four groups. Group 1 included 13 provenances (IN, CN, IL, MO, AL, PA, TN, GA, AR, IA, OK, MS/Al and MD), which had the largest CPL (3.875 cm), LW (3.028 cm), LA (15.112 cm2) and LPM (15.312 cm). Group 2 included the NC and MS provenances, which had the highest CLL/CLW (2.736 cm) and LC (77.194%). Group 3 contained WV, OH, KY and VA provenances, with the highest CLW (11.293 cm), LL (6.945 cm), Chl (36.096 SPAD) and PRO (40.042 μg·g-1). Group 4 comprised only one provenance, KS, which presented the highest CLL (29.709 cm), LL/LW (2.783), LP (9.917), LN (20.834), PA (64.192°) and Spro (1445.861 μg·ml-1). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Cluster analysis based on leaf phenotypic and physiological traits of 20 R. pseudoacacia L. provenances. https://doi.org/10.1371/journal.pone.0262278.g003 In addition, based on the results of the principal component analysis, systematic clustering analysis was performed on the eigenvectors of all 16 parameters in the four principal components using the same method described above (Fig 4). The results showed that the sixteen indicators could be classified into four groups when the euclidean distance was set to 23: (1) LP, LN, CLL/CLW and Chl; (2) Spro, PRO and LL/LW; (3) LL, LPM, LA, CLW, LW and CLL; and (4) LC and LA. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. Cluster diagram of 16 test indicators of 214 samples. https://doi.org/10.1371/journal.pone.0262278.g004 Mantel’s test based on sixteen trait parameters The correlations between the euclidean distance and geographical distance of the 13 phenotypic parameters and 3 physiological parameters of the 20 R. pseudoacacia provenances were analyzed. The results are shown in Fig 5 and indicate no significant correlations between the tested traits and the geographical distance of R. pseudoacacia at either leaf phenotypic or physiological levels. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. Relationships between geographical distance and tested trait distance for R. pseudoacacia L. (A: leaf phenotypic trait level, B: physiological level). https://doi.org/10.1371/journal.pone.0262278.g005 Elite tree selection based on different breeding goals According to the above analysis, 13 leaf phenotypic traits showed abundant variation in each provenance, which is helpful for breeding improved R. pseudoacacia plants for ornamental use and food production. Because of the significant positive correlations between CLL, LA, and LN, these three characteristics could also reflect the quantity and quality of R. pseudoacacia leaves. Therefore, when the above three traits were combined for respect to food-based breeding objectives, 40 elite trees were selected after the 214 accessions were analyzed (Table 6). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 6. Selection of superior trees from R. pseudoacacia L. geographical populations based on 13 leaf phenotypic traits. https://doi.org/10.1371/journal.pone.0262278.t006 Analysis of the three physiological traits, especially Spro and PRO, provided the basis for the selection of high-quality resistance resources for R. pseudoacacia. Firstly, based on the evaluation of PRO, sixty-three excellent trees (29.439%) were selected. Secondly, based on the evaluation of Spro, eighty-four excellent trees (39.252%) were selected, whose Spro content was approximately 1.5 times that in the original provenances. Lastly, based on the evaluation of PRO and Spro, thirty excellent trees were obtained (14.019%), whose PRO and Spro contents were 1.8 times those of the original provenances, and the other two indicators also increased compared with those of the original provenances, indicating that the stress resistance of these elite trees increased at the original population level (Table 7). S3 and S4 Tables in S1 File list the specific names of these elite trees (S3 and S4 Tables in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 7. Selection of superior trees from R. pseudoacacia L. geographical populations based on 3 physiological indexes. https://doi.org/10.1371/journal.pone.0262278.t007 Analysis of leaf phenotypic characteristics and physiological indicators Table 3 shows the mean values and the multiple comparison results based on Duncan’s test. Compound leaves and leaflets differed between the 20 provenances. Between the four compound leaf traits, there was no significant difference in compound leaf length (CLL), and the length ranged from 26.426 cm (CN) to 29.710 cm (IL). Compared with those at the other locations, the accessions in MS, with the largest value of 2.916 cm, showed a significant difference in CLL/CLW traits. Of the nine leaflet characteristics, there were no significant differences in petiole angle (PA) between accessions in LA. The accessions in MS had the smallest LL and LP, whereas those in KS had the largest LP and LN, with averages of 9.971 and 20.833, respectively. Similarly, the results for the multiple comparison tests of the three physiological traits showed no significant difference in proline content between provenances, but both the chlorophyll content and soluble protein content had significant differences. Among the above indicators, the provenances in IN, KS, and OH showed the highest values of certain characteristics—38.22±0.69 (Chl-SPAD value), 1445.861±899.893 μg·ml-1 (Spro) and 44.649±11.401 μg·g-1 (PRO), respectively. However, the provenances in KS, AR, and OK had the lowest values—31.98±0.54 (Chl-SPAD value), 1445.861±899.893 μg·ml-1 (Spro), and 19.257±3.532 μg·g-1 (PRO), respectively. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Differences in leaf phenotypic traits of 20 R. pseudoacacia L. provenances. https://doi.org/10.1371/journal.pone.0262278.t003 Differentiation coefficient analysis of sixteen trait parameters The analysis results (S1 Table in S1 File, Fig 1) showed that the coefficient of variation in the 13 leaf phenotypic traits among the different provenances was 3.741%-19.599%. All traits were lowest in LC and highest in LA, which indicates that LC has the smallest dispersion degree and highest stability among the leaf phenotypic traits; the case of LA is completely opposite. The coefficient of variation of the compound leaf traits was 11.236% and close to that of the leaflet traits (11.301%). At the provenance level, the total coefficient of variation was 11.281%, with the lowest and highest coefficients of variation at 7.286% (MS/AL) and 14.788% (TN), respectively, which shows that the provenance in TN had the most abundant leaf phenotypic diversity among the 20 black locust provenances. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Distribution patterns of coefficients of variation of 16 tested traits of R. pseudoacacia L. in 20 provenances. https://doi.org/10.1371/journal.pone.0262278.g001 Similarly, the total average variation of the three physiological parameters was 30.993%, and the value of soluble proteins (Spro) (42.363%) had the highest level, followed by the proline (PRO) (42.356%) and chlorophyll content (Chl) (8.260%); Spro had the highest stability of these parameters, as revealed by the maximum-to-minimum ratios of each indicator—6.364%, 8.735% and 19.178%, respectively. Of the 20 provenances, KY provenance had the highest total average coefficient of variation (52.786%), which was approximately 3.8 times that for the MS provenance, which presented the smallest variation coefficient. In conclusion, the variation richness of the 16 test indexes of the 214 black locust accessions was as follows: Spro (42.363%)>PRO (42.356%)>LA (19.599%)>PA (17.317%)>CLW (12.359%)>CPL (11.179%)>LL (10.942%)>LPM (10.741%)>CLL (10.708%)>CLL/CLW (10.697%)>LP (10.558%)>LW (10.522%)>LN (9.832%)>LL/LW (8.462%)>Chl (8.260%)>LC (3.741%). Similarly, the variation in richness at each provenance level was as follows: KY (18.514%)>WV (18.411%)>VA (17.323%)>OH (17.222%)>TN (17.105%)>CN (16.231%)>PA (16.174%)>IN (15.317%) = IL (15.317%)>GA (15.275%)>KS (14.943%)>MS (14.632%)>IA (14.606%)>MO (14.098%)>MD (13.852%)>OK (13.304%)>AL (13.298%)>AR (13.127%)>MS/AL (10.571%)>NC (10.224%). Analysis of the differentiation coefficients of sixteen trait parameters The differentiation coefficient and variance components among/within provenances ranged from 25.843% (CLW) to 71.655% (PA) at the leaf phenotypic trait level, with an average of 48.829%. The mean leaf phenotypic differentiation coefficient of the nine leaflet traits was 52.259%, lower than that of four compound leaf indicators (41.112%). Thus the differentiation level for the leaflets was slightly higher than for the compound leaves. In terms of single traits, the differentiation coefficients of CPL, LW, LP, LN and PA were larger than those within provenances, indicating that the variation among provenances was higher than within them (Table 4). Furthermore, the total mean leaf phenotypic differentiation coefficient among provenances was lower than within provenances, suggesting that the main variation of black locust occurred intra-provenance, not inter-provenance variation. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 4. Variance percentages and differentiation coefficients of 16 traits among/within provenances of R. pseudoacacia L. https://doi.org/10.1371/journal.pone.0262278.t004 Homoplastically, the variation of the three physiological parameters showed that Spro had the lowest and PRO, the highest differentiation coefficient at—23.956% and 49.137%, respectively. The total mean differentiation coefficient of the three indicators was 40.383%, but it was 59.617% within provenances. These results are consistent with the findings of leaf phenotypic traits, further demonstrating the importance of individual variation in black locust trees. Correlation analysis of sixteen trait parameters Sixty-one (50.833%) significantly correlated traits were identified by Pearson correlation analyses of the 16 parameters (CLL, CLW, CLL/CLW, CPL, LL, LW, LL/LW, LA, LPM, LC, LP, LN, PA, Chl, Spro, and PRO) in 20 R. pseudoacacia provenances (P<0.05 or P<0.01) (S2 Table in S1 File, Fig 2). Among these traits, there were 50 significantly correlated leaf phenotypic traits (P<0.05). For compound leaf traits, CLL/CLW showed a highly significant negative correlation with CLW (P<0.01) but was not significantly correlated with CPL (P>0.05). In terms of leaflets, LC showed significant negative correlations with LW, LL/LW, and LPM, and LW was significantly positively correlated with LA, LPM, and LC (P<0.01). However, CLL/CLW showed significant negative correlations with LL, LW, LA, LPM, and LP (P<0.01). There were 10 significantly correlations between phenotypic and physiological traits. CPL and LW exhibited extremely significant negative correlations between all pairs of physiological traits; Chl presented significant positive correlations with LP and LN, and PRO was significantly negatively correlated with CLW and LA (P<0.05). There were no significant correlations between physiological traits, except for a correlation between Spro and PRO. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 2. Heat map of a correlation analysis of 16 parameters from 214 R. pseudoacacia L. samples. Notes: *, P<0.05; **, P<0.01. https://doi.org/10.1371/journal.pone.0262278.g002 Principal component analysis of sixteen trait parameters Five principal components explained the investigated characteristics with eigenvalues greater than 1.0 (Table 5), and the cumulative contribution rate reached 80.004%, essentially reflecting the main information contained in all the measured indexes. In principal component 1, the CLL, CLW, CPL, LL, LW, LA, and LPM with relatively high absolute values and were representative of some compound leaf and leaflet characteristics; in principal component 2, LP and LN were relative; CLL/CLW, LL/LW and LC constituted the principal component 3; and the principal component 4 emphasized PA, Spro, and PRO. The fifth component separated Chl. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 5. Principal component analysis of 16 traits of different R. pseudoacacia L. geographic provenances. https://doi.org/10.1371/journal.pone.0262278.t005 Cluster analysis based on sixteen trait parameters Cluster analysis was performed for twenty different R. pseudoacacia provenances using the method of hierarchical cluster analysis between groups, taking sixteen trait parameters as variables (Fig 3). When the euclidean distance was set to 10, the twenty R. pseudoacacia provenances could be divided into four groups. Group 1 included 13 provenances (IN, CN, IL, MO, AL, PA, TN, GA, AR, IA, OK, MS/Al and MD), which had the largest CPL (3.875 cm), LW (3.028 cm), LA (15.112 cm2) and LPM (15.312 cm). Group 2 included the NC and MS provenances, which had the highest CLL/CLW (2.736 cm) and LC (77.194%). Group 3 contained WV, OH, KY and VA provenances, with the highest CLW (11.293 cm), LL (6.945 cm), Chl (36.096 SPAD) and PRO (40.042 μg·g-1). Group 4 comprised only one provenance, KS, which presented the highest CLL (29.709 cm), LL/LW (2.783), LP (9.917), LN (20.834), PA (64.192°) and Spro (1445.861 μg·ml-1). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 3. Cluster analysis based on leaf phenotypic and physiological traits of 20 R. pseudoacacia L. provenances. https://doi.org/10.1371/journal.pone.0262278.g003 In addition, based on the results of the principal component analysis, systematic clustering analysis was performed on the eigenvectors of all 16 parameters in the four principal components using the same method described above (Fig 4). The results showed that the sixteen indicators could be classified into four groups when the euclidean distance was set to 23: (1) LP, LN, CLL/CLW and Chl; (2) Spro, PRO and LL/LW; (3) LL, LPM, LA, CLW, LW and CLL; and (4) LC and LA. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 4. Cluster diagram of 16 test indicators of 214 samples. https://doi.org/10.1371/journal.pone.0262278.g004 Mantel’s test based on sixteen trait parameters The correlations between the euclidean distance and geographical distance of the 13 phenotypic parameters and 3 physiological parameters of the 20 R. pseudoacacia provenances were analyzed. The results are shown in Fig 5 and indicate no significant correlations between the tested traits and the geographical distance of R. pseudoacacia at either leaf phenotypic or physiological levels. Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 5. Relationships between geographical distance and tested trait distance for R. pseudoacacia L. (A: leaf phenotypic trait level, B: physiological level). https://doi.org/10.1371/journal.pone.0262278.g005 Elite tree selection based on different breeding goals According to the above analysis, 13 leaf phenotypic traits showed abundant variation in each provenance, which is helpful for breeding improved R. pseudoacacia plants for ornamental use and food production. Because of the significant positive correlations between CLL, LA, and LN, these three characteristics could also reflect the quantity and quality of R. pseudoacacia leaves. Therefore, when the above three traits were combined for respect to food-based breeding objectives, 40 elite trees were selected after the 214 accessions were analyzed (Table 6). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 6. Selection of superior trees from R. pseudoacacia L. geographical populations based on 13 leaf phenotypic traits. https://doi.org/10.1371/journal.pone.0262278.t006 Analysis of the three physiological traits, especially Spro and PRO, provided the basis for the selection of high-quality resistance resources for R. pseudoacacia. Firstly, based on the evaluation of PRO, sixty-three excellent trees (29.439%) were selected. Secondly, based on the evaluation of Spro, eighty-four excellent trees (39.252%) were selected, whose Spro content was approximately 1.5 times that in the original provenances. Lastly, based on the evaluation of PRO and Spro, thirty excellent trees were obtained (14.019%), whose PRO and Spro contents were 1.8 times those of the original provenances, and the other two indicators also increased compared with those of the original provenances, indicating that the stress resistance of these elite trees increased at the original population level (Table 7). S3 and S4 Tables in S1 File list the specific names of these elite trees (S3 and S4 Tables in S1 File). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 7. Selection of superior trees from R. pseudoacacia L. geographical populations based on 3 physiological indexes. https://doi.org/10.1371/journal.pone.0262278.t007 Discussion Plant variation is closely related to the genetic characteristics of plants and their growth environment. In general, the larger the distribution range of a tree species, the greater the variation, and the smaller the distribution range, the smaller the variation [46, 47]. Morphological variation is an important part of genetic variation; the greater the area in which a tree species is distributed, the larger its genetic variation, leaf phenotypic and physiological differences [48, 49]. Analysis of the variation characteristics of the leaf phenotypic and physiological characteristics of different Robinia pseudoacacia L. provenances In this study, different R. pseudoacacia provenances were planted at the same site to reduce environmental variation. Our results showed that all black locust traits measured in the field varied among the provenances. Thirteen leaf phenotypic traits and 3 physiological indexes for black locust showed significant differences among 20 different provenances. Among all provenances, the TN provenance presented the largest coefficient of variation of phenotypic traits. And the KY provenance presented the largest coefficient of variation of physiological indexes of all and also the maximum of all 16 tested parameters. The LA, PA, and Spro in the KS provenance were largest, conversely, its CLL/CLW, LL/LW, LC, LP, LN, Chl, and PRO were the smallest. The degree of variation among different traits within the one provenance was diverse, indicating an imbalance in the degree of variation of leaf phenotypic traits and physiological indexes between different provenances of R. pseudoacacia. At the leaf trait level, different R. pseudoacacia provenances exhibited significant differences, consistent with the results of Granata et al. regarding Acer campestre leaf area morphological characteristics [50]. At the physiological level, compared to Spro and PRO, Chl had the smallest variation coefficient, with maximum-to-minimum ratios of 5.167% (Spro vs Chl) and 5.128% (PRO vs Chl). The large difference between these different physiological indicators may be due to the data obtained by the portable SPAD-502 Plus chlorophyll meter. Factors such as plant variety (genotype), environmental conditions, planting density, and nutrient conditions can affect SPAD values [51]. Using a spectrophotometric method compared to the portable chlorophyll meter SPAD-502 method, a study by Wang et al. showed a high coefficient of variation of the main greening tree species in China’s northwestern Liaoning Province [52]. It is commonly held that a coefficient of variation of traits greater than 10% represents a large difference between individuals and equates to a rich variation in traits [53]. In our research, the coefficients of variation of 13 indexes were higher than 10%, with an average total coefficient of variation of 17.924%, showing abundant variation in leaf phenotypic and physiological indexes of black locust trees, which is the basis of species selection. This variability is consistent with the findings of previous studies of the national R. pseudoacacia fine variety bases in Ji, Shanxi Ji, which showed a rich diversity in leaf phenotypic traits among 96 genotypes [38]. In addition, the degree of variation of LC and PRO was relatively large, and the degree of variation of LC was small, indicating that the same traits were affected differently in the same habitat or that different traits were affected in the same habitat. This may be related to the black locust’s the inherent genetic factors and to the influence of environmental factors [46, 54]. The differentiation coefficient of the tested indexes in our results showed that the main variation of black locust were intra-provenance variation. This is consistent with the results of previous studies on black locust [43] and other plant species [55, 56], which reveals a large potential for the selection of individual variation and can provide potential opportunities for black locust genetic improvement and germplasm preservation. Correlation coefficients can be used to reveal the relationships between measured traits and thus greatly influence selections as part of breeding strategies [57, 58]. For all the tested parameters, compound leaf and leaflet traits generally revealed moderate and strong relationships, respectively. In particular, relationships between LA, LPM, and CLL as well as between CLW, LL, and LW have been are significantly positively correlated in most studies, such as those involving Salix psammophila [59] and Phoebe bournei [60]. Moreover, LL, LW, LL/LW, LA, LPM and LP were significantly negatively correlated with CLL/CLW, indicating leaflet traits were greatly affected by compound leaf shape. For black locust, comprehensive assessment of compound and leaflet traits were able to truly respond to the phenotypic traits of leaves. In addition, leaf size and shape can effectively reflect changes in the plant’s natural environment and adjust morphologically to water evaporation and heat loss to adapt to the environment. The CLL/CLW, LL/LW, LA, LMP, and LC traits of black locust leaves could also reflect the adaptation to the growth environment to a certain extent. For the above reasons, most leaf traits of black locust exhibited complex relationships. Among the three physiological indexes, PRO had a significant positive correlation with Spro, which may be due to the Chl value obtained by the chlorophyll meter SPAD-502 instead of values to values obtained spectrophotometrically. Principal component analysis is a multivariate technique widely used for dimension reduction, that is, analyzing multiple related variables of one or a few comprehensive indicators [61]. In our research, leaf traits and physiological parameters were analyzed by principal component analysis. The cumulative contribution rate of the first three principal components was 65.033%, which was lower than that for the R. pseudoacacia germplasm in Shanxi and Phoebe bournei. Possible explanations for these results could be the weak correlations between these traits and/or the differences in the number and types of measured parameters [3, 38, 60]. Furthermore, from eigenvalue and variance contribution rate, traits such as compound length, width, petiole size, leaflet length, leaflet size, leaflet circumference and leaflet area are the main factors in the phenotypic difference of black locust samples. And Srpo and PRO are the main factors in the physiological difference of black locust samples. The above traits can be focused on in the actual breeding. Mantel’s test of phenotypic and physiological characteristics of different Robinia pseudoacacia L. provenances Cluster analysis is a mathematical method used to find similarities between measured indexes/materials used in a group by revealing the real categories of the population and reducing the number of data points [62]. Multiple test parameters were therefore divided into dominant groups by cluster analysis, the most common and most effective classification method. The sixteen indicators could be classified into four groups, as shown in Fig 4. After comprehensive evaluations were performed, group II mainly reflected physiological indexes, and groups I and III mainly reflected leaf phenotypic characteristics of R. pseudoacacia leaves. Indicators I, II and III were the preferred test indicators to achieve the breeding objectives for practical production applications, including stress resistance, ornamental value and food production. Our results are consistent with those of many previous studies on morphological variation in Paeonia rockii. Four categories were divided based on 12 fruit traits, and group II was screened to maximize the economic yield per plant [63]. The results of Mantel’s test were similar to the clustering results based on euclidean distance, revealing a nonsignificant geographical variation pattern of phenotypic and physiological parameters. This is consistent with the results of previous studies of not only black locust via molecular markers such SSRs [43] and ISSRs [64] and via allozymes [65, 66] but also of other tree species, such as Prosopis alba [67]. Possible explanations for these results are that the collection range of provenances is broad while the numbers are small and/or black locust has migrated to its present-day range more recently. Elite tree selection of different Robinia pseudoacacia L. provenances Phenotypic changes following a change in natural selection are particularly important for undergoing continuous adaptation [68]; this reflects the ability of plants to grow normally in nature and indicates an ability to protect the environment. Excellent tree selection is a basic method for genetic improvement of forest trees; this method involves selecting individuals with relatively good comprehensive leaf phenotypic traits after comparisons with other trees under the same site conditions [69]. Some tree breeding programs aim to select a group of black locust trees that could be used to improve ornamental quality, tolerance to soil infertility and food production for livestock. For practical applications in the present study, forty and thirty elite trees were selected according to their aggregate indicators, that were significantly correlated (18.692% and 14.019% of the total sample for three-leaf phenotypic traits (CLL, LA, and LN) and two physiological indexes (PRO and Spro), respectively. The selection rate in our study was lower than that of a comprehensive scoring method for selecting Taxodium distichum by Wang et al. [70]. Different elite trees were selected based on different evaluation indexes, methods and breeding objectives, resulting in different selection rates. Analysis of the variation characteristics of the leaf phenotypic and physiological characteristics of different Robinia pseudoacacia L. provenances In this study, different R. pseudoacacia provenances were planted at the same site to reduce environmental variation. Our results showed that all black locust traits measured in the field varied among the provenances. Thirteen leaf phenotypic traits and 3 physiological indexes for black locust showed significant differences among 20 different provenances. Among all provenances, the TN provenance presented the largest coefficient of variation of phenotypic traits. And the KY provenance presented the largest coefficient of variation of physiological indexes of all and also the maximum of all 16 tested parameters. The LA, PA, and Spro in the KS provenance were largest, conversely, its CLL/CLW, LL/LW, LC, LP, LN, Chl, and PRO were the smallest. The degree of variation among different traits within the one provenance was diverse, indicating an imbalance in the degree of variation of leaf phenotypic traits and physiological indexes between different provenances of R. pseudoacacia. At the leaf trait level, different R. pseudoacacia provenances exhibited significant differences, consistent with the results of Granata et al. regarding Acer campestre leaf area morphological characteristics [50]. At the physiological level, compared to Spro and PRO, Chl had the smallest variation coefficient, with maximum-to-minimum ratios of 5.167% (Spro vs Chl) and 5.128% (PRO vs Chl). The large difference between these different physiological indicators may be due to the data obtained by the portable SPAD-502 Plus chlorophyll meter. Factors such as plant variety (genotype), environmental conditions, planting density, and nutrient conditions can affect SPAD values [51]. Using a spectrophotometric method compared to the portable chlorophyll meter SPAD-502 method, a study by Wang et al. showed a high coefficient of variation of the main greening tree species in China’s northwestern Liaoning Province [52]. It is commonly held that a coefficient of variation of traits greater than 10% represents a large difference between individuals and equates to a rich variation in traits [53]. In our research, the coefficients of variation of 13 indexes were higher than 10%, with an average total coefficient of variation of 17.924%, showing abundant variation in leaf phenotypic and physiological indexes of black locust trees, which is the basis of species selection. This variability is consistent with the findings of previous studies of the national R. pseudoacacia fine variety bases in Ji, Shanxi Ji, which showed a rich diversity in leaf phenotypic traits among 96 genotypes [38]. In addition, the degree of variation of LC and PRO was relatively large, and the degree of variation of LC was small, indicating that the same traits were affected differently in the same habitat or that different traits were affected in the same habitat. This may be related to the black locust’s the inherent genetic factors and to the influence of environmental factors [46, 54]. The differentiation coefficient of the tested indexes in our results showed that the main variation of black locust were intra-provenance variation. This is consistent with the results of previous studies on black locust [43] and other plant species [55, 56], which reveals a large potential for the selection of individual variation and can provide potential opportunities for black locust genetic improvement and germplasm preservation. Correlation coefficients can be used to reveal the relationships between measured traits and thus greatly influence selections as part of breeding strategies [57, 58]. For all the tested parameters, compound leaf and leaflet traits generally revealed moderate and strong relationships, respectively. In particular, relationships between LA, LPM, and CLL as well as between CLW, LL, and LW have been are significantly positively correlated in most studies, such as those involving Salix psammophila [59] and Phoebe bournei [60]. Moreover, LL, LW, LL/LW, LA, LPM and LP were significantly negatively correlated with CLL/CLW, indicating leaflet traits were greatly affected by compound leaf shape. For black locust, comprehensive assessment of compound and leaflet traits were able to truly respond to the phenotypic traits of leaves. In addition, leaf size and shape can effectively reflect changes in the plant’s natural environment and adjust morphologically to water evaporation and heat loss to adapt to the environment. The CLL/CLW, LL/LW, LA, LMP, and LC traits of black locust leaves could also reflect the adaptation to the growth environment to a certain extent. For the above reasons, most leaf traits of black locust exhibited complex relationships. Among the three physiological indexes, PRO had a significant positive correlation with Spro, which may be due to the Chl value obtained by the chlorophyll meter SPAD-502 instead of values to values obtained spectrophotometrically. Principal component analysis is a multivariate technique widely used for dimension reduction, that is, analyzing multiple related variables of one or a few comprehensive indicators [61]. In our research, leaf traits and physiological parameters were analyzed by principal component analysis. The cumulative contribution rate of the first three principal components was 65.033%, which was lower than that for the R. pseudoacacia germplasm in Shanxi and Phoebe bournei. Possible explanations for these results could be the weak correlations between these traits and/or the differences in the number and types of measured parameters [3, 38, 60]. Furthermore, from eigenvalue and variance contribution rate, traits such as compound length, width, petiole size, leaflet length, leaflet size, leaflet circumference and leaflet area are the main factors in the phenotypic difference of black locust samples. And Srpo and PRO are the main factors in the physiological difference of black locust samples. The above traits can be focused on in the actual breeding. Mantel’s test of phenotypic and physiological characteristics of different Robinia pseudoacacia L. provenances Cluster analysis is a mathematical method used to find similarities between measured indexes/materials used in a group by revealing the real categories of the population and reducing the number of data points [62]. Multiple test parameters were therefore divided into dominant groups by cluster analysis, the most common and most effective classification method. The sixteen indicators could be classified into four groups, as shown in Fig 4. After comprehensive evaluations were performed, group II mainly reflected physiological indexes, and groups I and III mainly reflected leaf phenotypic characteristics of R. pseudoacacia leaves. Indicators I, II and III were the preferred test indicators to achieve the breeding objectives for practical production applications, including stress resistance, ornamental value and food production. Our results are consistent with those of many previous studies on morphological variation in Paeonia rockii. Four categories were divided based on 12 fruit traits, and group II was screened to maximize the economic yield per plant [63]. The results of Mantel’s test were similar to the clustering results based on euclidean distance, revealing a nonsignificant geographical variation pattern of phenotypic and physiological parameters. This is consistent with the results of previous studies of not only black locust via molecular markers such SSRs [43] and ISSRs [64] and via allozymes [65, 66] but also of other tree species, such as Prosopis alba [67]. Possible explanations for these results are that the collection range of provenances is broad while the numbers are small and/or black locust has migrated to its present-day range more recently. Elite tree selection of different Robinia pseudoacacia L. provenances Phenotypic changes following a change in natural selection are particularly important for undergoing continuous adaptation [68]; this reflects the ability of plants to grow normally in nature and indicates an ability to protect the environment. Excellent tree selection is a basic method for genetic improvement of forest trees; this method involves selecting individuals with relatively good comprehensive leaf phenotypic traits after comparisons with other trees under the same site conditions [69]. Some tree breeding programs aim to select a group of black locust trees that could be used to improve ornamental quality, tolerance to soil infertility and food production for livestock. For practical applications in the present study, forty and thirty elite trees were selected according to their aggregate indicators, that were significantly correlated (18.692% and 14.019% of the total sample for three-leaf phenotypic traits (CLL, LA, and LN) and two physiological indexes (PRO and Spro), respectively. The selection rate in our study was lower than that of a comprehensive scoring method for selecting Taxodium distichum by Wang et al. [70]. Different elite trees were selected based on different evaluation indexes, methods and breeding objectives, resulting in different selection rates. Conclusions The present study showed the following: (1) LC and LA exhibited opposite variations at the phenotypic level, of which LC had the highest stability; (2) when the differentiation coefficients of four compound leaves, nine leaflets and three physiological traits were compared the differentiation level of leaflet traits was higher than that of the other two types of indexes; (3) the variation of test traits is mainly attributed to differences within provenances, although the variation between provenances could not be ignored; and (4) there was a nonsignificant geographical variation pattern of phenotypic and physiological parameters. Suggestions and elite tree resources for germplasm preservation strategies and efficient breeding are provided to preserve the genetic resources of R. pseudoacacia. Supporting information S1 File. https://doi.org/10.1371/journal.pone.0262278.s001 (DOCX)