- Open Access
- Open Peer Review
Validity and reliability of a food frequency questionnaire for assessing dietary intake among Shanghai residents
Nutrition Journal volume 18, Article number: 30 (2019)
Few localized food frequency questionnaires (FFQ) have been developed and used in Chinese nutrition surveys despite China’s large population and diverse dietary habits.
We analyzed data collected in two waves (six months apart) of the Shanghai Diet and Health Study in 2012–2013, from 1623 Shanghai residents (798 men and 825 women) older than 18 years. The results of 3-day 24-h dietary recalls (HDR) plus condiment weighing were used to evaluate the validity and reliability of the SDHS FFQ.
The median and first and third quartiles for energy intake (in kcal) derived from the FFQ1 and FFQ2 were 1566.5 (1310.1–1869.6) and 1561.9 (1280.2–1838.4), respectively, of which protein (in g) was 54.3 (42.5–65.8) and 52.9 (42.4–64.5), fat (in g) was 49.8 (37.2–64.7) and 47.9 (34.9–61.9), and carbohydrates (in g) was 227.3 (180.8–277.9) and 228.1 (182.2–275.2) in the reliability analysis. The median and first and third quartiles for energy-intake differences between the FFQ1 and the 3-day 24-HDR with condiment weighing was 59.3 (− 255.5–341.6), of which protein was − 5.2 (− 18.7–7.8) and fat was − 11.2 (− 30.8–5.3). The adjusted Spearman’s correlations were 0.33–0.77 for validity and 0.46–0.79 for reliability. The intra-class correlation coefficients exceeded 0.46 (validity) and 0.47 (reliability) for macronutrient intake. The consistency between the same and adjacent quartiles was approximately 80% for various nutrients.
The reliability and comparative validity of the SDHS FFQ is similar to FFQs that are used worldwide.
Although 24-h dietary recall (24-HDR) and dietary records with weighed foods have been used to measure usual dietary intake, the resource burden and economic constraints of these methods make them unsuitable for most large-scale studies [1, 2]. Short-term recalls and dietary records are expensive and unrepresentative of usual intake and therefore, inadequate for the assessment of past dietary intake. The food frequency questionnaire (FFQ) is the most commonly used instrument to assess past dietary intake in epidemiological studies on the relationship between dietary factors and diseases, primarily because of its low cost and ability to capture usual dietary patterns .
Frequency data can explain much of the variation in dietary intake, and FFQs can provide sufficient accuracy to rank individuals in terms of risks for subsequent health outcomes. FFQs have been used in many studies to predict associations between dietary intake and disease-specific mortality and morbidity . However, the performance of an FFQ depends on its sensitivity to cultures, ethnic backgrounds, geographical areas and differences in study populations. Thus, the validity and reliability of FFQs should be evaluated for use with diverse samples.
In 2012, a population-based cohort study in Shanghai, China was initiated. The Shanghai Diet and Health Study (SDHS) was designed and implemented by the government to investigate the nutritional status of Shanghai’s residents and food contaminants in Shanghai. It also aimed to examine how these factors affect health outcomes, given Shanghai’s rapid economic development and the introduction of considerable variations in diet and eating habits that may influence food intake [5,6,7,8]. The SDHS proposal included assessments of dietary intake of the cohort members at the baseline survey (spring, summer, fall and winter, 2012–2013), and every 5–10 years afterwards. The purpose of the population-based cohort study was to collect data on the dietary intake and diet transitions of Shanghai residents.
The FFQ that was used previously in different geographical areas for the National Nutrition and Health Survey was also used in Shanghai . However, substantial dietary discrepancies among different areas due to different dietary habits and traditions were found. Therefore, the SDHS developed a localized FFQ based on similar FFQs and local dietary habits. At baseline and follow-up, we collected dietary data using both the 24-HDR plus condiment weighing and FFQs in four waves (seasons). The aim of this research was to evaluate the validity and reliability of a 134-item quantitative FFQ. The instrument was designed to capture the usual dietary intake of Shanghai participants, and therefore, consisted of foods commonly consumed by Shanghai residents. The validated FFQ can provide an alternative way to capture dietary intake in future follow-ups and it can be used in related studies.
The SDHS is an ongoing open-cohort study that was initiated in 2012. It was designed and implemented by the Shanghai government as a prospective examination of food consumption, energy and nutrient intake, and food contaminants in Shanghai, and their effects on the health of its residents. Given Shanghai’s rapid economic growth , the consumption patterns and eating habits of Shanghai residents have undergone great changes, which are likely to influence food intake and health outcomes. The study’s design was explained in the published literature [5,6,7,8]. Four waves of the SDHS were implemented during its first 2 years. Data were collected during May–June 2012 (spring), August–September 2013 (summer), November–December 2012 (fall) and January–February 2013 (winter). All the participants completed a 3-day 24-HDR and condiment weighing and an FFQ in each wave (season) [5,6,7,8].
Development of the FFQ
The SDHS’s FFQ was developed using similar dietary questionnaires from several epidemiological studies conducted in Shanghai, the 2010–2013 nation-wide nutrition survey conducted in China [10,11,12,13,14,15,16] and updated information from Shanghai’s 2010–2011 dietary survey. The FFQ was designed for use by trained interviewers to collect information from Shanghai adults about their dietary intake during the previous 3 months. It is a 134-item quantitative FFQ with three parts consisting of supplementary questions regarding eating-out frequency, cooking oil and condiments, and eating habits. A total of 123 food items and food groups are included in the questionnaire, which represent approximately 95% of the most commonly consumed foods in Shanghai in 2011.
The FFQ food groups include staples, beans, vegetables, fungus, algae, fruit, dairy foods, meat, aquatic products, eggs, sweets and snacks, beverages and condiments. First, the participants were asked to report whether they consumed each food and food group. If they responded yes, they were asked to state how many times per day, week, month or year. The interviewer also asked participants about their average amount of intake for each item, each time it was consumed. The interviewers read aloud the standard portion size of each food item for every question. Visual aids relating to the standard portion sizes were shown to the participants.
Data pertaining to oils and condiments were obtained by inquiring about how many grams of the oils and other condiments were consumed by the entire family during the previous month, and how many family members consumed the condiments at home.
Dietary validation analysis
Data from a sample of 1623 participants older than 18 years of age from the SDHS were included in the validity study. They completed the 134-item FFQ and a 3-day 24-HDR plus condiment weighing in all four waves (hereafter, “3-day 24-HDR plus condiment weighing” will be referred to as “24-HDR” for concision). Household condiment consumption (such as edible oils, salt, sauces, etc.) was determined by weighing all food consumed by the household over three consecutive days. Three-day 24 h recalls were done on three consecutive days to match with the weighing. It was determined by examining changes in inventory from the beginning to the end of each day, in combination with a weighing and measuring technique. All condiments remaining after the last meal before initiation of the survey were weighed and recorded. All purchases and wasted condiments were also recorded. At the end of the survey, all remaining condiments were again weighed and recorded. Data from two seasons (approximately 6 months apart) were used in this validity study. Wave 1 was conducted from January to February in 2013 and wave 2 from August to September 2013.
The nutrient database for the FFQ was developed in accordance with the constituent ratio of the amount of each food listed in the same item using representative 24-HDR data. For example, the citrus fruit group included oranges, tangerines, pomelos, and citrus gonggan. First, we calculated the amount of each food in the food group based on data obtained in the 24-HDR during each of the four waves of the study. Based on the constituent ratio of the amount of the top 10 foods weighted by their constituent ratios, the converted food composition database was recalculated for each food group.
The frequency of food intake was converted to the number of times consumed “per day” and multiplied by the amount of intake (g) to obtain the daily dietary intake of each food group. The amounts of oils and other condiments reported in the FFQ for the entire family was divided by the number of family members, and then, divided by the proportion of meals consumed at home, and converted to daily intake. The amount of the food intake per day was entered into the FFQ nutrient database. Daily totals for energy and nutrients were calculated, followed by macronutrient intake as a percentage of energy.
Data obtained from the 24-HDRs and the weighing of condiments were converted to the amount of each food item consumed per person per day. Then, the data on nutrients from the 24-HDR were analyzed using a food composition table developed by the Chinese Nutrition Society.
Medians and interquartile ranges were calculated for all nutrients because the data for most of them were not distributed normally. The Wilcoxon signed-rank test was used to examine absolute differences between the FFQ and 24-HDR and the FFQ1 and FFQ2. Reproducibility was evaluated using all participants’ data from both of the FFQs by comparing the two rounds, using adjusted Spearman’s correlations. Comparative validity was assessed using adjusted Spearman’s correlations, and the Bland-Altman analysis  was used to examine differences between the FFQ and the 24-HDR. To examine the FFQ’s reliability and validity, participants were classified into quartiles based on the distributions of the data on their energy and macronutrient intake from the results of the FFQ and the reference method. Similar proportions of participants were classified into the same, adjacent or extreme quartiles. Correlations were used to detect linear relationships between the variables. Bland-Altman plots are used to evaluate agreement between two different measurements to determine the precision of one method compared with a reference method, in this study, to compare the FFQ with the 24-HDR. Quartile agreement was also used to assess quartile consistency. The results were considered statistically significant at a 0.05 level (two-tailed). Statistical analyses were performed with SAS software, version 9.4 (SAS Institute Inc., Cary, NC).
General characteristics of the participants (e.g., age, marital status, occupation, education, weight, family income, and region) are presented in Table 1. Data from 1623 participants, including 798 men and 825 women, were analyzed. More than half of the men and 0.6% of the women were current smokers. Alcohol consumption was reported by 33.7% of the men and 5.2% of the women.
The median, first, and third quartiles for the energy and macronutrient intake were estimated using two FFQs and 24-HDRs. Differences between the FFQ1 and 24-HDR are presented in Table 2. The median and first and third quartiles for energy intake (kcal) derived from the FFQ1 and FFQ2 were 1566.5 kcal (1310.1–1869.6 kcal), and 1561.9 kcal (1280.2–1838.4 kcal), respectively, of which protein was 54.3 g (42.5–65.8 g) and 52.9 g (42.4–64.5 g), fat was 49.8 g (37.2–64.7 g) and 47.9 g (34.9–61.9 g), and carbohydrates was 227.3 g (180.8–277.9 g) and 228.1 g (182.2–275.2 g) in the reproducibility analysis. In general, there was no significant difference between waves 1 and 2 of the FFQs. The median and first and third quartiles for energy-intake differences between the FFQ1 and the 24-HDR was 59.3 kcal (− 255.5–341.6 kcal), of which protein was − 5.2 g (− 18.7–7.8 g) and fat was − 11.2 g (− 30.8–5.3 g).
The adjusted Spearman’s correlations for reproducibility ranged from 0.46 to 0.79. The consistency between the same and adjacent quartiles was 80%. The intraclass correlation coefficient (ICC) between the FFQ1 and FFQ2 were 0.59, 0.71, 0.48, 0.47 for energy and macronutrients intake, and ranged from 0.34 to 0.72for micronutrient intake (Table 3).
Comparative validity was evaluated by calculating the correlation coefficients for the FFQ and 24-HDR, as shown in Table 4. The adjusted Spearman’s correlations for validity ranged from 0.33 to 0.77. The consistency between the same and adjacent quartiles was 80%.
The median, and first and third quartiles for energy and macronutrient intake were estimated using the two FFQs and 24-HDRs. The validity and reproducibility, as measured by correlations are summarized in Tables 5 and 6. The validity and the reproducibility of most of the energy and nutrient intake among the men were higher than that of the women. When the data were analyzed by age group, we found the highest reproducibility in the group that was less than 45 years old; however, the highest validity was found in the group that was older than 60 years of age (Additional file 1).
The results of the Bland-Altman analyses of the energy, protein, fat and carbohydrate intake are shown in Figs. 1, 2, 3 and 4. Differences in intake between the FFQ and the 24-HDR are plotted on the Y-axis and the mean intake derived from the two tools is presented on the X-axis. The mean difference and 95% lower and upper limits for energy intake was 31.9 kcal (− 891.3–827.5 kcal), of which protein intake was − 7.3 g (− 45.6–30.8 g), fat was − 23.7 g (− 86.6–39.1 g) and carbohydrate was 54.9 g (− 85.1.3–195 g).
The results indicate that the SDHS FFQ has comparative validity and reliability to the 24-HDR, and can be used to categorize major nutrients to determine their intake with relative accuracy among residents of Shanghai. This report describes the validity and reproducibility of a FFQ designed to capture Shanghai residents’ usual intake of nutrients. The reference method was the 24-HDR (i.e., 3-day 24-HDR and household condiment weighing), which was conducted at the beginning and the end of a 6-month period. We evaluated the performance of the FFQ by comparing the intake of nutrients reported using this instrument with the intake obtained using the 24-HDR.
The 24-HDR has been used in most nutrition studies in China . The method provides accurate estimates of study participants’ usual dietary intake. However, this method is usually expensive, resource-intensive, and it yields only information collected over a short period . FFQs have a lower respondent burden, are relatively inexpensive, do not require trained interviewers, and can be semi-automated using technology, making them practical for large epidemiological studies [20, 21]. Our study used a large representative population to test the validity and reliability of the FFQ by comparing it with the 24-HDR method. The FFQ was found to be useful in the nutrition survey.
In comparison with reproducibility and validity studies on other FFQs in the same population, we observed relatively higher correlation coefficients, indicating good reproducibility of our FFQ. Villegas et al. and Shu et al. have reported acceptable validity and reproducibility of an FFQ for the assessment of energy and nutrient intake in the Shanghai Men’s Health Study (SMHS) and Shanghai Women’s Health Study (SWHS). The ranges for Pearson’s correlations for validity and reproducibility were 0.33–0.58 and 0.38–0.53, respectively, for the SMHS, and 0.41–0.66 and 0.30–0.59, respectively, for the SWHS. The SMHS and SWHS reported differences between the FFQ and 24-HDR that ranged from − 21.3 to 31.8% and − 8.8 to 12.1, respectively [13, 15]. Our FFQ was developed based on the latest dietary data of Shanghai residents and the most frequently used FFQs in nation-wide surveys , whereas the FFQs used in the SMHS and SWHS were developed based on the most commonly consumed foods in urban Shanghai in 1996. Due to significant changes in dietary intake, China’s food supply has become much more diverse and “off-season” foods can be found in homes during most times of the year in developed cities, such as Shanghai. Our FFQ provides a better representation of the food items and food groups recently found in Shanghai. Moreover, our study collected data on oils, salt and other condiments in our FFQ, which provided dietary data that are more comprehensive than those reported in other studies. Subar et al.  reported scores ranging from 0.41 to 0.83 on the Diet History Questionnaire, 0.19 to 0.80 on the Block FFQ and 0.28 to 0.83 on the Willett FFQ using measurement error models adjusted for energy with the 24-HDR as the reference method. The findings from our study are consistent with their results.
Assuming the data obtained from the 24-HDR is close to participants’ “true” intake, we found that the SDHS FFQ can provide a better estimate of macronutrients, including energy and the contributions of protein, fat and carbohydrates, and other nutrients, such as cholesterol, pantothenate, choline, folate, Vitamin E, Biotin and Vitamin K. However, our FFQ did not perform as well on estimates of thiamine, zinc and niacin, which may be due to seasonal food differences between the FFQ and the 24-HDR.
The strength of our study is that we included more than 1.6 thousand randomly selected residents from all districts of Shanghai to provide the best representation of food intake of the Shanghai population compared to any other study conducted in the same area. Second, we developed our FFQ based on the latest local dietary survey and other FFQs in China with similar objectives to reveal the actual intake of this population. Third, we added condiment-related questions to our FFQ, which have not been analyzed with other samples. This change should provide dietary data that are more comprehensive than other FFQs. Fourth, we originally collected data four times during each of the four seasons and they were found to have validity and reliability for all four seasons. We found comparative validity and reliability among the seasons (Additional file 1: Tables S1-S4). This is the first study to examine the validity and reliability of a FFQ over four seasons. We assessed the FFQ’s validity and reliability by gender and age group and found differences among these groups.
This study has some limitations. First, the FFQ has been validated only as an interviewer-administered FFQ but not as a self-report questionnaire, Second, the FFQ evaluated in our study is a localized data-collection instrument that may be used to assess the local diets of populations from Shanghai or southeast China. Moreover, the results are not necessarily transferable to other populations because of regional variations in local foods [23, 24]. Each region of China should develop a localized FFQ in accordance with their specific dietary habits and traditions instead using a uniform FFQ from nationwide surveys. Validity and reproducibility may vary by gender and age; therefore, researchers should carefully recruit diverse groups. In conclusion, this study indicates that the SDHS FFQ can reliably and accurately measure the usual intake of major nutrients among residents of Shanghai.
The reliability and comparative validity of the SDHS FFQ is similar to FFQs that are used worldwide.
24-h dietary recalls
Food frequency questionnaires
Shanghai Diet and Health Study
The Shanghai Men’s Health Study
Shanghai Women’s Health Study
Grandjean AC. Dietary intake data collection: challenges and limitations. Nutr Rev. 2012;70(Suppl 2):S101–4. https://doi.org/10.1111/j.1753-4887.2012.00545.x.
M B. 24-hour dietary recall and food record methods. In: Willett W, editor. Nutritional epidemiology. 2nd ed. Oxford: Oxford University Press; 1998.
Willett W. Food-frequency methods. In: Willett W, editor. Nutritional epidemiology. 2nd ed. Oxford: Oxford University Press; 1998.
Nishida C, Uauy R, Kumanyika S, Shetty P. The joint WHO/FAO expert consultation on diet, nutrition and the prevention of chronic diseases: process, product and policy implications. Public Health Nutr. 2004;7(1A):245–50.
Zhu ZN, Lu Y, Wu CF, Zou SR, Liu H, Wang CF, Luo BZ, Yu HT, Mi M, Wang GQ, Xiong LB, Wang WJ, Luo CY, Zang JJ, Wang ZY, Jia XD, Feng XG, Guo CY, Wu F. General plan of Shanghai diet and health survey. Zhonghua liu xing bing xue za zhi = Zhonghua liuxingbingxue zazhi. 2018;39(7):876–9. https://doi.org/10.3760/cma.j.issn.0254-6450.2018.07.003.
Zang J, Luo B, Wang Y, Zhu Z, Wang Z, He X, Wang W, Guo Y, Chen X, Wang C, Guo C, Zou S, Jia X, Wu F. Eating out-of-home in adult residents in Shanghai and the nutritional differences among dining places. Nutrients. 2018;10(7). https://doi.org/10.3390/nu10070951.
Zang J, Guo C, Wang Z, Cheng Y, Jin W, Zhu Z, Zou S, Wang C, Lu Y, Wang W, He X, Tang H, Gao H, Jia X, Wu F. Is adherence to the Chinese dietary guidelines associated with better self-reported health? The Chinese dietary guidelines adherence score. Asia Pac J Clin Nutr. 2018;27(4):914–24. https://doi.org/10.6133/apjcn.072018.02.
Zang J, Yu H, Zhu Z, Lu Y, Liu C, Yao C, Bai P, Guo C, Jia X, Zou S, Wu F. Does the dietary pattern of Shanghai residents change across seasons and area of residence: assessing dietary quality using the Chinese diet balance index (DBI). Nutrients. 2017;9(3). https://doi.org/10.3390/nu9030251.
Zhang B, Zhai FY, Du SF, Popkin BM. The China health and nutrition survey, 1989-2011. Obes Rev. 2014;15(Suppl 1):2–7. https://doi.org/10.1111/obr.12119.
Dai Q, Shu XO, Jin F, Potter JD, Kushi LH, Teas J, Gao YT, Zheng W. Population-based case-control study of soyfood intake and breast cancer risk in Shanghai. Br J Cancer. 2001;85(3):372–8. https://doi.org/10.1054/bjoc.2001.1873.
Ji BT, Chow WH, Yang G, McLaughlin JK, Zheng W, Shu XO, Jin F, Gao RN, Gao YT, Fraumeni JF Jr. Dietary habits and stomach cancer in Shanghai, China. Int J Cancer. 1998;76(5):659–64.
Shu XO, Gao YT, Yuan JM, Ziegler RG, Brinton LA. Dietary factors and epithelial ovarian cancer. Br J Cancer. 1989;59(1):92–6.
Shu XO, Yang G, Jin F, Liu D, Kushi L, Wen W, Gao YT, Zheng W. Validity and reproducibility of the food frequency questionnaire used in the Shanghai Women's health study. Eur J Clin Nutr. 2004;58(1):17–23. https://doi.org/10.1038/sj.ejcn.1601738.
Shu XO, Zheng W, Potischman N, Brinton LA, Hatch MC, Gao YT, Fraumeni JF, Jr. (1993) A population-based case-control study of dietary factors and endometrial cancer in Shanghai, People's Republic of China. Am J Epidemiol 137 (2):155–165.
Villegas R, Yang G, Liu D, Xiang YB, Cai H, Zheng W, Shu XO. Validity and reproducibility of the food-frequency questionnaire used in the Shanghai men's health study. Br J Nutr. 2007;97(5):993–1000. https://doi.org/10.1017/S0007114507669189.
Zheng W, Blot WJ, Shu XO, Gao YT, Ji BT, Ziegler RG, Fraumeni JF Jr. Diet and other risk factors for laryngeal cancer in Shanghai, China. Am J Epidemiol. 1992;136(2):178–91.
Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet (London, England). 1986;1(8476):307–10.
Chang J, Wang Y (2016) Chinese Nutrtion and health surveillance 2010–2013. Beijing.
Hoffmann K, Boeing H, Dufour A, Volatier J, Telman J, Virtanen M. Estimating the distribution of usual dietary intake by short-term measurements. Eur J Clin Nutr. 2002;56(Suppl 2):S53–62.
Cade J, Thompson R, Burley V, Warm D. Development, validation and utilisation of food-frequency questionnaires - a review. Public Health Nutr. 2002;5(4):567–87. https://doi.org/10.1079/phn2001318.
RS G. Principles of nutritional assessment. 2nd ed. NY: Oxford University Press, New York; 2005.
Subar AF, Thompson FE, Kipnis V, Midthune D, Hurwitz P, McNutt S, McIntosh A, Rosenfeld S. Comparative validation of the block, Willett, and National Cancer Institute food frequency questionnaires : the Eating at America's Table Study. Am J Epidemiol. 2001;154(12):1089–99.
Cade JE, Burley VJ, Warm DL, Thompson RL, Margetts BM. Food-frequency questionnaires: a review of their design, validation and utilisation. Nutr Res Rev. 2004;17(1):5–22. https://doi.org/10.1079/NRR200370.
Plummer M, Kaaks R. Commentary: an OPEN assessment of dietary measurement errors. Int J Epidemiol. 2003;32(6):1062–3.
The authors thank the investigators of the District Centers for Disease Control and Prevention, and the Community Center for their contributions.
The current study was supported by the National Nature Science Foundation of China (No. 81602851), the Outstanding Young Talents in Medical and Health Sciences in Shanghai (No. 2017YQ043), and the Fourth Round of the Three-year Public Health Program and Key Disciplines (No.15GWZK0801).
Availability of data and materials
The datasets generated and/or analyzed during the current study are not publicly available due to the confidentiality regulations of our agency, but are available from the corresponding author upon request.
Ethics approval and consent to participate
The SDHS was approved by the Ethical Review Committee of the Shanghai Municipal Centers for Disease Control and Prevention. All the participants were fully informed of the purpose and procedures of the study and signed written consent forms before they enrolled in it.
Consent for publication
The study and our institution did not receive payments or services from a third party (government, commercial, private foundation, etc.) for any aspect of the submitted work (including but not limited to grants, data monitoring board, study design, manuscript preparation, statistical analysis, etc.) at any time. There are no relevant conflicts of interest. We have no financial relationships (regardless of amount of compensation) with entities, as described in the journal’s instructions. There are no relevant conflicts of interest. We have no patents, no matters planned, pending or issued, that are broadly relevant to the work or any other relationships/conditions/circumstances that might present a potential conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Table S1. Median and 25–75th percentile(Q1-Q3) of energy and nutrient intake of FFQs among seasons in Shanghai Diet and Health Study. Table S2. Median and 25–75th percentile(Q1-Q3) of energy and nutrient intake of 24-HDR among seasons in Shanghai Diet and Health Study. Table S3. Validity and reliability of the FFQs in different seasons. Table S4. Validity and reliability of the FFQs in different subgroups. (DOCX 39 kb)