This article has Open Peer Review reports available.
Analysis of dietary interventions. A simple payoff matrix for display of comparative dietary trials
- Richard D Feinman^{1}Email author,
- Eugene J Fine^{1, 2} and
- Jeff S Volek^{3}
https://doi.org/10.1186/1475-2891-7-24
© Feinman et al; licensee BioMed Central Ltd. 2008
Received: 25 June 2008
Accepted: 29 August 2008
Published: 29 August 2008
Abstract
Objective
To provide a simple method for presentation of data in comparative dietary trials.
Methods
Individual data from each diet are ranked and all possible paired comparisons are made and displayed in a pay-off matrix which can be color-coded according to the magnitude of the differences between the two diets. Probability of outcome can be calculated from the fraction of matrix elements corresponding to specified conditions. The method has the advantage of emphasizing differences and providing the maximum amount of information.
Results
The method was tested with values from the literature and allows intuitive sense of the comparative effectiveness of the two diets. In a test case in which a cross-over study had been performed the matrix derived from theoretical paired comparisons (treating the data as two parallel studies) was consistent with the results from the actual pairing in the cross-over.
Conclusion
The matrix method is a simple way of providing access to the differences between dietary trials. It exaggerates differences but can be used in combination with group statistics that, conversely, provide reliability at the expense of detailed information.
Background
It is increasingly recognized that individual variation is an important consideration in dietary interventions for weight loss, cardiovascular disease and other conditions. Position papers of scientific organizations make the point that it is "unlikely that a single strategy is best for all" (e.g. [1]). It seems, however, that the principle is more widely applied in the breach than in the observance and most dietary trials continue to report means and standard deviations or other measures of group statistics. The use of group statistics in a dietary intervention implies not only that the population is uniform but, in addition, that there is a best diet for all individuals. In fact, given the difficulty in effecting weight loss, the test of a dietary strategy might better be whether it can be made to work for anybody. In this sense, the outliers become the key parameters in diet and the probability of reaching some criterion, rather than average behavior is intuitively a useful measure of efficacy of a diet. Further, weight loss involves inconvenience and dedication and a prospective dieter or physician is sensibly asking about relative payoffs of different dietary approaches – in essence, which diet to bet on. Thus, what one wants from a dietary comparison is to be able to compare individual performances on different diets rather than group values. Non-parametric statistics based on ranking such as the Mann-Whitney tests are applicable but because the rank becomes the primary variable, information is lost that a potential dieter would like to know.
Here we describe a simple way to present individual data in comparative dietary trials that preserves the maximum amount of information and allows judgments of efficacy to be made. The idea is based on assessment of all possible paired comparisons and deduction of benefit from evaluation of a pay-off matrix, analogous to a matrix comparing different outcomes in games of strategy [2].
The experimental ideal for taking account of individual variation is the cross-over protocol, that is, paired variables, in which subjects alternate between the two different diets. In this case, an odds-ratio for specified differential performance or some similar parameter will give the information that a dieter might want. Because it is difficult to perform any dietary intervention, however, a cross-over experiment is not always feasible and one may have to compare two groups in parallel who cannot generally be assumed to have uniform responses to diet.
Methods
Individual results for the two diets to be compared are ranked according to outcome, for example, weight loss. All possible differences are calculated and these differences constitute the elements of a payoff matrix. The matrix elements can be color coded to indicate particular levels of relative payoff, e.g. > 2 kg difference in weight loss. Probabilities of particular outcomes can also be calculated. The underlying rationale is that, since we don't know which subject in intervention A should be matched with which subject in intervention B, we consider all such possibilities and consider all the outcomes. The method has the advantage that it does not assume any particular distribution. It has the disadvantage that it assumes that the sample distribution of responses is somehow representative of the population distribution, and it therefore tends to over-emphasize differences.
The matrix presentation brings out the qualitative features of individual responses and allows a graphic representation of the comparative effects. It emphasizes differences and therefore is exaggerated in comparison with sample means which tend to emphasize consistency.
We illustrate the method with data from a paper by Volek, et al. [3]. In this cross-over study, 15 men and 13 men were assigned to a low fat (LF) diet or a very low carbohydrate ketogenic diet (VLCKD). After a certain period subjects were switched to the other diet. We first treat the data as if this were a parallel experiment, that is, as if the LF and the VLCKD group were separate individuals. We then ask how the results compare to the actual outcomes where the pairings are known.
Probability of differences in outcomes in diet comparisons
PROBABILITY | |||
---|---|---|---|
MATRIX | X-OVER | ||
MEN | VLCKD > LF | 0.73 | 0.73 |
LF > VLCKD | 0.27 | 0.27 | |
VLCKD – LF > 2 | 0.59 | 0.53 | |
LF – VLCKD > 2 | 0.17 | 0.27 | |
VLCKD – LF > 5 | 0.31 | 0.33 | |
LF – VLCKD > 5 | 0.02 | 0 | |
WOMEN | VLCKD > LF | 0.78 | 0.62 |
rLF > VLCKD | 0.21 | 0.31 | |
VLCKD – LF > 2 | 0.69 | 0.54 | |
LF – VLCKD > 2 | 0.09 | 0.31 | |
VLCKD – LF > 5 | 0.09 | 0.31 | |
LF – VLCD > 5 | 0 | 0 |
The matrix display shows that, beyond average performance, those subjects who benefited more from carbohydrate restriction compared to reduced fat generally did so in a big way (> 5 kg differential weight loss). Table 1 summarizes the probabilities of the various theoretical outcomes. An individual dieter is not guaranteed better outcome on either diet but the table demonstrates that they have a far better chance of showing dramatic results if they go with the VLCKD.
Volek's experiment [3] had a second phase in which subjects switched diets allowing a test of the matrix method, that is, one can ask how the actual within-subject comparisons compare to the anticipated outcomes of the matrix presentation. Table 1 compares the probabilities from the matrix analysis and from the data in the cross-over experiment. The table shows that there is generally good agreement – fortuitously good for the overall comparison of the two groups – and it is clear that the matrix predicts, and only slightly exaggerates, the value of the VLCKD compared to LF.
Discussion
Group statistics tend to destroy information in order to gain reliability. The case might be made that not every experiment should have the same standards for the relative importance of these two parameters. In experiments where people can reasonably be expected to have similar responses, such as drug trials, an expectation value based on average outcome is the most relevant. On the other hand, given the difficulty of staying on any diet, the high non-Gaussian prevalence of obesity in the general populations [4] and the variability due to hidden variables (like non-exercise activity thermogenesis (NEAT, [5]), the prospective dieter is really asking which diet to bet on, that is, what is the best possible payoff for each diet and what are the odds, analogous to picking a strategy in the theory of games [2]. In the end, nobody loses an average amount of weight and the frequently quoted conclusion that low-carbohydrate and LF diets are the same at one year (e.g., [6]) might be further enlightened by individual analysis along the lines suggested from a matrix presentation.
Group statistics are integrative methods and decrease the signal to noise ratio and increase reliability but, because they obscure differences can repress future experiments. The matrix method is a derivative method and therefore increases the noise. It has reduced reliability but because it highlights potential differences, can provide a guide to future experiment and is hypothesis generating. Outlier data can be compared against prospective trials using Chi-square or Fischer exact testing to improve the reliability of the original observation. Use of both methods would provide maximum information about the nature of the comparison. Finally, while the example given is a weight loss experiment, the method is generalizable to all experimental parameters, such as lipid profile, that may be under the control of diet.
Comparison of matrix data from the results of multiple experiments will be required to make the most reliable conclusions. Along these lines, a comparison of the numerical differences between LF and low carbohydrate diets, regardless of statistical significance showed the likelihood that the carbohydrate-restricted diet was better [7], that is, treated as separate Bernoulli trials, the results were not likely to be random. The matrix method would demonstrate whether the numerical benefit of the low carbohydrate diet was due to a sub-group with unusually good performance.
Declarations
Authors’ Affiliations
References
- Klein S, Sheard NF, Pi-Sunyer X, Daly A, Wylie-Rosett J, Kulkarni K, Clark NG: Weight management through lifestyle modification for the prevention and management of type 2 diabetes: rationale and strategies: a statement of the American Diabetes Association, the North American Association for the Study of Obesity, and the American Society for Clinical Nutrition. Diabetes Care. 2004, 27 (8): 2067-2073. 10.2337/diacare.27.8.2067.View ArticlePubMedGoogle Scholar
- Williams JD: The Compleat Strategyst. Being a Primer on the Theory of Games of Strategy. 1986, New York , DoverGoogle Scholar
- Volek JS, Sharman MJ, Gomez AL, Judelson DA, Rubin MR, Watson G, Sokmen B, Silvestre R, French DN, Kraemer WJ: Comparison of energy-restricted very low-carbohydrate and low-fat diets on weight loss and body composition in overweight men and women. Nutr Metab (Lond). 2004, 1 (1): 13-10.1186/1743-7075-1-13.View ArticleGoogle Scholar
- Ogden CL, Yanovski SZ, Carroll MD, Flegal KM: The epidemiology of obesity. Gastroenterology. 2007, 132 (6): 2087-2102. 10.1053/j.gastro.2007.03.052. 2007/05/15View ArticlePubMedGoogle Scholar
- Levine JA, Kotz CM: NEAT--non-exercise activity thermogenesis--egocentric & geocentric environmental factors vs. biological regulation. Acta Physiol Scand. 2005, 184 (4): 309-318. 10.1111/j.1365-201X.2005.01467.x. 2005/07/20View ArticlePubMedGoogle Scholar
- Foster GD, Wyatt HR, Hill JO, McGuckin BG, Brill C, Mohammed BS, Szapary PO, Rader DJ, Edman JS, Klein S: A randomized trial of a low-carbohydrate diet for obesity. N Engl J Med. 2003, 348 (21): 2082-2090. 10.1056/NEJMoa022207.View ArticlePubMedGoogle Scholar
- Fine EJ, Feinman R, Isasi C, Wylie-Rosett J: A new systematic review of low carbohydrate diets. Obesity Research. 2004, 12 (Suppl): 223P-Google Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.