6. Discussion and conclusion

6.1 The importance of five independent SMTIs and their features comparison in five aspects

As depicted in Tab. 3, two auxiliary SMTIs - CM and DCM - are unable to evaluate independently, which need support from five independent SMTI, and MCM refers to a method combining two to several SMTIs, which also relies on one to several independent SMTIs among five, therefore the five independent SMTIs is the cornerstone of QES in both ESGB and EEWH. In addition, the composition of the five SMTIs in two GBRTs’ QESs directly decides the quality of QES to a large extent, since the significant differences of the five SMTIs’ quality in several aspects is not hard to observe and analyzed, shown as Tab. 8.

Thus, the features and qualities of five independent SMTIs in five aspects were analyzed and discussed initially, so as to facilitate the further discussion on QESs of two GBRTs, and the five aspects embraces 1) the quantified capacity; 2) the objective and scientific degree of evaluation; 3) the adaptability of complex indicators; 4) the ability to cover the number of evaluation dimensions; and 5) the expressive simplicity. As described in Tab. 8, a basic tendency is not hard to extract, which is the quality of SMTIs in five aspects mentioned above are gradually improved from DSM, MOM, IAM to FSM generally.

6.2 The unique of ASM among five independent SMTIs

It is noteworthy that ASM is out of this general trend, since its unique characteristics. ASM totally relies on the subjective judgments from a limited number of professional people with preference and bias, thus, it is assessment result seems subjective and lack of data support. In order to mitigate the negative impact of subjective evaluation, in EEWH, much effort was conducted: 1) strict evaluation process was regulated to the assessment by ASM, which should be identified by an assessment committee composed of experts in this field[22]; 2) the number of TI adopted ASM is also limited to a small proportion (merely approximately 10％); 3) most importantly, a score threshold was set for each TI evaluated by ASM in EEWH. Apparently, the ASM adopted in EEWH drew lessons from the Delphi Method so as to improve ASM’s reliability. The Delphi Method was developed during the 1950s by workers from the RAND Corporation to acquire the best available opinion consensus from a group of experts[23,24], and Delphi Method was primarily employed at where judgmental information is indispensable[24,25].

Although much works need to be done in order to make up the shortcoming of ASM, it is hard to deny its unique value: making up the system defect of GBRTs. Reportedly, none of the GBRTs can cover whole passive design strategies and update frequently to catch up the development of novel technologies, thus it is essential to reserve some flexibility in GBRTs[22]. In this respect, ASM can match up with this flexible requirement well, since the assessment of the other four independent SMTIs (DSM, MOM, IAM, and FSM) relies on the clear evaluation criteria, and, obviously, it is hard to predict the forthcoming technologies and integrate them into GBRTs with clear description. Hence, leaving some evaluation space for the uncertain evaluation items is a wisdom choice, and ASM is able to assess those unknown items more efficiently and accurately relative to DSM, MOM, IAM, and FSM, based on experts’ up-to-date experience and knowledge.

As mentioned above, none of the GBRTs is perfect, thus, normally, the innovation indicators group were set to make up for this inevitable deficiency. Interesting, there is a discrepancy between two GBRTs’ QES associated with the assessment of innovation indicators: DSM and ASM are employed by ESGB and EEWH correspondingly. Obviously, ASM is more suitable for innovation evaluation relative to DSM, since that the innovation is unknown when the GBRT was established, hence, using DSM, it is hard to illustrate the sealed innovation definitely and clearly, which directly leads to GBRT is unable to guide practically the evaluation works well. In contrast, using ASM, the continuously updated knowledge from experts may ensure the evaluation of innovation better.

6.3 The quantification and mature degree of four SMTIs

On the foundation of section 6.1, the discrepancies of five independent SMTIs will be discussed further in this section. As the discussion in section 6.2, ASM is a special one, thus, the following analysis scope excludes it and merely focus on the differences of DSM, MOM, IAM, and FSM. In addition, the five dimensions of features of five independent SMTIs (shown in Tab. 8) can be represented as quantitative and mature degrees.

Based on this limitation, the development trend, related to quantitative and mature degree, reflected in Tab. 8 is obvious and can be transformed into Fig. 5, which depicts the continuous development stages among those four independent SMTIs. To be specific, DSM, MOM and IAM, and FSM are located at initial, intermediate and advanced stages respectively. The higher stage means the SMTI is more mature and can quantitatively evaluate TIs more efficient, scientific and accurate.

6.4 Conclusion

From the comparative study of section 5.3, it was confirmed that DSM and FSM dominate the ESGB and EEWH’s QES correspondingly. In addition, section 6.4 has revealed that DSM and FSM belong to the initial stage and advanced stage respectively, which means FSM is the most mature SMTI and is capable to quantitative assess most efficiently and accurately, oppositely, DSM at the other end. Integration of the information mentioned above, it is not hard to conclude an interesting finding: The QES of EEWH is mature than ESGB. Two reasons, at least, can support this deduction: 1) EEWH’s SMTI diversity, especially in independent SMTI, is rich than ESGB (Tab. 7), which means the QES of EEWH can adapt to TIs with variously different characteristics better; 2) EEWH’s QES mainly relies on the most advanced SMTI (FSM), rather than DSM, a relatively immature SMTI, compared to ESGB’ QES with highest utilization proportion of DSM (63.9％). In addition, this deduction is in line with fact that EEWH established QES in 2005 with 4 times revision till now relative to ESGB, which just set up QES in 2014 without any update (Tab. 1).

7. Suggestions

Four main improvement suggestion listed as below were proposed for ESGB.

1. Regarding to the evaluation of innovation indicators, ASM is recommended to ESGB. In addition, refer to the Delphi Method and the regulation of EEWH’ ASM, a strictly standardized assessment process and the score threshold are suggested to the employment of ASM in ESGB.

2. It is recommended to increase the diversity of SMTI in QES of ESGB, such as MOM and FSM, so as to adapt a variety of characteristics of different TIs and achieve a more reliable evaluation result.

3. FSM is strongly recommended to be frequently utilized by QES of ESGB, instead of DSM, especially for the evaluation of complicated TIs with numerous impact factors, since the quantitative assessment of FSM is more reliable than DSM, as shown in Tab. 8 and Fig. 5. In addition, MOM and IAM is also a better option compared to DSM.

4. FSM relies on corresponding databases to a considerable degree, thus it is also suggested that those databases should be established or enhanced as soon as possible so as to meet this potential requirement 

分享让更多人看到