“if you torture the data enough, nature will always confess” – Ronald Coase

Introduction

Recently a debate took place on the pages of Indian Express (starting on July 7 ,2023 edition) about the quality of survey-based data released by the National Sample Survey Organizations (NSSO). Before that, doubts about quality of Indian official statistics data and methodology used for estimating important social developmental indicators were aired through two working papers issued by Economic Advisory Council of the Prime Minister (EAC-PM). When a couple of eminent economists of impeccable credentials raises serious doubt about the quality and accuracy of some key statistical products that are supposed to reflect, directly or indirectly, the performance of the economy, it is incumbent on the statistical professionals to take these criticisms seriously and put out their non-partisan views to the public at large

The present article aims to examine the veracity of the criticisms raised by two members of the EAC-PM regarding the quality of estimation of key policy performance indicators by NSSO surveys, who in the past have used NSSO survey data on Employment and Unemployment (Gundimeda, Sanyal & Others, 2006) and NSSO data on Morbidity (Ravi 2016) without indicating any reservation about data quality and sampling design of NSSO surveys.

The statistical issues have been raised by the following two working papers of EAC-PM.

1. Reversing the Gaze:  Re-examining Estimates of India’s Development Indicators by International Organisation:  EAC-PM Working Paper Series EAC-PM/WP/14/2023; March 2023 Authors: Sanjeev Sanyal, Aakanksha Arora, and Srishti Chauhan; henceforth, referred to as WP-1.
2. Assessing the National Surveys for its Representativeness:   An Analysis of the Data Quality of the National Sample Survey (NSS)- July 2023 Authors: Shamika Ravi, Mudit Kapoor, and S V Subramanian; henceforth, referred to as WP-2.

Section I:  Criticisms of methodologies adopted for computation of different indicators in WP-1

This working paper deals with accuracy and soundness of “three widely used data-driven development indicators” which according to the authors “seem to stagnate or even deteriorate despite a rise in per capita income in India” (Executive summary pp3). These three indicators are” Childhood Stunting”, “Female Labour Force Participation Rate (FLFPR)”, and “Life Expectancy at Birth”.

Indicator No-1 Stunting

Two important measures of deleterious effect of childhood malnutrition are  wasting and stunting. Wasting is defined as low weight-for-height. Stunting is defined as low height-for-age. The prevalence of stunting amongst Indian children in the age group of 0-5 years has been declining over the years. The authors have quoted the following trend- 35.5% in 2019-21(rural 37.3%; urban 30.1%)- down from 38.4% In 2015-16 and 48% in 2005-06. The source of this data is National Family Health Survey (NFHS). The estimates of stunting have been worked out using the WHO’s global standard for this measure. The authors have questioned the methodology adopted by WHO in deriving this standard and, furthermore, its applicability in the Indian context.

Issues raised:

1. The authors of WP-1 are of the view that given India’s economic growth, it is highly unlikely that “over one-third of Indian children suffer from Stunting”.
2. Physical status like height of people vary significantly between and within any geographical region. It is well known that even “the best-fed Indian child” is shorter than the average children from the developed countries and so they are not comparable. Any international standard, therefore, is most likely to overestimate prevalence of stunting amongst Indian children below 5 year of age.
3. Like the children, Indian mothers are “substantially shorter than their counterpart” from the developed countries. The authors have approvingly quoted a paper by Kalsson et al. (2021) in which, after factoring in maternal height, “India’s stunting prevalence was revised drastically downwards from 38% to 25% using Demographic and Health Survey data 2019-21.” (all quotes above are from WP-1)

The above issues cannot be critically analysed unless we have a clear understanding of what is being measured and how. According to WHO, children are stunted if their length/height is below (-)2 standard deviation (SD) from the WHO Child Growth Standards median for the same age and sex. It should be noted that the standard is for “Growth” and not actual height of children of a particular reference population. It then follows that the criticism at point no 2 above is invalid because it is referring to actual heights of children of different reference population. In fact, the objective of the study that WHO conducted between 1997 and 2003 for arriving at the required standard is clear from the title of the study- Multicentre Growth Reference Study (MGRS) (de Onis et al 2004).

The critics may still argue that the choice of the 6 countries, namely, Brazil, Ghana, India, Norway, Oman, and the United States, for this study was arbitrary and cannot be considered a representative sample of the world at large. According to Mercedes de Onis, Study Coordinator for MGRS, “sampling frame was selected on the basis of scientific and health advocacy considerations” (De Onis 2006 ). To validate this sampling strategy and to evaluate the appropriateness of pooling data for the purpose of constructing a single international growth standard, an ANOVA analysis of MGRS data was carried out to measure the contribution to total volatility in linear growth as between sites and within sites. The result of this analysis clearly points out that between-sites volatility contributes only 3% to the total volatility of growth measure.  The study concludes – “these analyses document the strong similarity in linear growth from birth to 5 y in major ethnic groups living under relatively affluent conditions. They also support the inclusion of all six MGRS sites for the purpose of constructing a single international standard.”

As regards choice of sites within a selected country, independent study of the chosen sites was carried out to “determine whether affluent population” in the chosen site had a growth performance like that in developed countries” (Bhandari Nita et al 2002, Bhandari Nita et al 2004). This study’s concluding observation is given in its title itself which reads as “Growth performance of affluent Indian children is similar to that in developed countries”.

The last point of criticism refers to the impact of maternal height on a child’s height and thereby on the required measure of prevalence of stunting among their children.  Omar Karlsson in his 2021 paper has proposed a new measure called “maternal height-standardized prevalence of stunting (SPS)” as another measure of stunting and compared it with the standard measure of stunting used in WHO studies. Karlsson has defined SPS as follows:

Where cps is the crude prevalence of stunting at each centimeter (cm) of maternal height in the sample and pd is the probability density for each cm of maternal height for the reference population. The result obtained by Karlsson et al. is critically dependent on their use of a specific probability density which is the distribution of mother’s height (measured in centimeter) .  The following pedagogical example proves the point

Let us assume of incidence of stunting is more among mothers of lesser heights and the number of children bore by such mothers are more than the rest. It would be then a simple arithmetic calculation to demonstrate, that factoring of maternal height in the way Karlsson et al have done, would lead to significant decline in the computed prevalence of stunting.

We assume two distinct heights of mothers.  The next assumption is that mothers of lesser heights, if such factoring of maternal height is recursively followed back, would be poor, bearing more child with stunted growth. Let the size of reference population of mother be one thousand with 30% of them being poor and having lesser height. This group of poor mothers have 600 children with age less than 5 and while the rich mothers have only 300 such children. I am assuming 80% of these poor children are suffering from stunting while 20% of the rich children are found to be stunted. Weighted average of prevalence of stunting will depend on which weights are used as can be seen below.

Weighted Average % of stunted children using share of each children type as weight = 60%

Weighted average % of stunted children using share of mother type as weight = 38%”

It is quite clear from the above example that reported decline of prevalence of stunting by factoring in maternal height is a false narrative. The concept of maternal-height adjusted prevalence of stunting, as described by Addo et al. (2013), indeed aims to capture intergenerational influences on growth and the extent to which children can attain their genetic height potential, especially in low- and middle-income countries. Statistically, the true measure of effect of mother’s height on a child’s growth would be an estimate of conditional probability of child being stunted given a height of mother.

Finally, there is no creditable evidence that by constructing country specific synthetic growth curve would lead to drastic change in measured prevalence of stunting. For a large country like India, with multiple ethnic groups with different mean height and weight of children at various age group, construction of a single standard for optimal growth of children will face the same issues that are now being raised against WHO standard. Some details about efforts that are underway to build country specific growth curve called Synthetic Growth Curve are provided in the end note[i].

In conclusion, a comparative position of India among some selected countries in terms of this indicator on stunting, per the WHO standard, should be of greater importance to policy makers.

see the footnote below

---

[i] A new “synthetic growth curve” has been proposed and applied to a small group of countries like Indonesia, Malwai and a few more. There is a confusion about the use of CDC growth curve in USA as against WHO growth curve. But CDC has clarified the purpose of introducing CDC growth reference curve is not to create a new standard (see Grummer-Strawn et al 2010).

 The idea of synthetic growth curve was proposed by Michael Hermanussen and others in a 2015 paper (Hermanussen Michael et al 2015). In the Indian context few researchers have sought to estimate prevalence of stunting using India specific reference growth curve using this methodology. All of them used methodology proposed by Hermanussen and his collaborators(see Khadilkar et al 2019 , Khadilkar 2025, and Mehta et al 2022 ). A perusal of growth curve construction methodology shows that the synthetic references were not for “optimal” growth but for average growth of selected age-groups.

End of footnote

Table 1.1: Trend in prevalence of stunting among children under age 5 for selected countries.

	Prevalence of stunting, height for age (% of children under 5)
Country	Starting year	Data for starting year	End Year	Data for end Year	Number of years between two data points	Average Rate of  Yearly Reduction
Bangladesh	2000	51.1	2019	28	19	-3.2%
Brazil	1996	13	2007	7	11	-5.6%
China	2000	17.8	2017	4.8	17	-7.7%
India	1999	54.2	2017	34.7	18	-2.5%
Indonesia	2000	42.4	2018	30.8	18	-1.8%
Malaysia	1999	20.7	2019	21.8	20	0.3%
Pakistan	2001	41.4	2018	37.6	17	-0.6%
South Africa	1999	30.1	2017	21.4	18	-1.9%
Sri Lanka	2000	18.3	2016	17.3	16	-0.4%
Vietnam	2000	43.2	2020	19.6	20	-4.0%

Indicator No 2. Female Labour Force Participation Rate (FLFPR)

The second section of WP1 is titled – “Flaws in Female Labour Force Participation Estimation: ILO’s Unworkable Maths”.  Research on this subject on low FLFPR of Indian female in rural areas has been underway since India’s independence. In the World Population Conference held at Belgrade in 1965, J.N.Sinha of India, suggested a U-type movement in FLFPR as per capita income increases “The rate’s decline with an increase in income up to $500, but begin to rise with further gains in income.  ..  Within the rural areas, the rates vary inversely with agricultural prosperity and the proportion of non-agricultural work force. Labour force participation of women also declines with literacy, but female education above the matriculation level favours higher rates of employment (Sinha 1965, see also Reddy D. Narasimha 1979).”

Bhalla and Kaur (2011) made one interesting extension to the computation of labour force by adding students in the age group 15-59 to the labour force. They called it adjusted labour force and demonstrated the varying trends of both ‘adjusted’ and ‘unadjusted’ FLFPR in rural and urban areas. (Table 2.1)

Table 2.1 Percentage of female population in Labour Force and adjusted labour Force

Area	Year->		1983	1993/94	1999/00	2004/05	2007/08
Rural	Unadjusted FLFPR		45.1	53.1	45.2	44.7	37.6
	Adjusted FLFPR		46.8	55.9	49.5	50.4	44.4
Urban	Unadjusted FLFPR		23	23	22.5	24.3	19.7
	Adjusted FLFPR		30.5	33.3	33.4	35.5	32.3

Table 2.2: Labour force participation rates (%), UPSS and augmented definition, by sex and area

	UPSS Definition					Augmented definition
Area	1994	2000	2005	2010	2012	1994	2000	2005	2010	2012
Rural-Female	49	45.4	49.4	37.8	35.8	80.8	77	76.3	70.1	66.8
Urban-Female	23.8	20.8	24.4	19.4	20.5	45.2	38.5	39.1	35.9	32.1
All areas-Female	42.7	38.9	42.7	32.6	31.2	71.8	66.8	66.4	60.3	56.4
Rural Male	87.6	85.3	85.9	82.5	81.3	87.9	85.3	86.2	82.8	81.6
Urban Male	80.1	78.7	79.2	76.2	76.4	80.2	78.7	79.3	76.3	76.4
All-areas Male	85.6	83.4	84	80.6	79.8	85.9	83.4	84.3	80.9	80

 

The expanded definition of labour force did not result in any substantial difference in the trend observed in the NSSO defined rural female labour force participation rate. This result of augmentation is not unexpected given the substantial share of persons engaged in domestic activities in rural India- see Table 2.3 below.

Table 2.3: Shares of the working-age population engaged in domestic duties.            

Gender	Region	1994	2000	2005	2010	2012
Female	Rural	42.4	43.9	39.8	49.4	49.9
Female	Urban	60.7	61.9	59.3	62.1	61.1
Male	Rural	0.4	0.4	0.4	0.5	0.5
Male	Urban	0.5	0.4	0.4	0.6	0.3

The ILO economists identified 4 key factors for changes in female participation rates- “increased attendance in education”, “increased household consumption levels”, changes in measurement of economic activities”, and “changes in employment opportunities” and carried out a statistical exercise to estimate the relative contribution of these factors towards the decline of FLFPR in rural areas. Their conclusion at the end is worth reproducing.

“The econometric results indicate that religion and social perceptions of women, women’s level of education, household size and income, and the presence of young children in the household all influence the likelihood of India’s women to participate in the labour market. We find that structural characteristics in the labour market have played a more important role than changes in the underlying characteristics of the female working-age population in influencing participation rates.” (Kapsos 2014, page 31, emphasis added).

Sonal Das & others carried out a regression analysis to identify determinants of female labor force participation in India in both urban and rural areas (Das Sonali 2015). Amongst the 5 “stylized facts” noted by the authors based on 5 NSSO EUS between 1993-94 and 2011-12, two are about causal factors for the observed declining trend in rural FLFPR:

• There is a U-shaped relationship between education and labor force participation rates of women
• Income has a dampening effect on female labor force participation rates

It is now beyond doubt that the issue of declining FLFPR in rural India is a known and well researched issue. Everyone who has dealt with the issue are aware of the differences in definitional boundary of “economic activity” adopted by NSSO surveys as compared to that of national income accounting.  The difference is spelt out in the 68^th round of NSSO EU survey.

Although production of any good for own consumption is considered as economic activity by UN System of National Accounts, production of only primary goods for own consumption was considered as economic activity by NSSO for the purpose of the survey. While the former considers activities like own account processing of primary products as economic activities, processing of primary products for own consumption was not considered as economic activity in the NSS surveys (NSS 68th Round Employment-Unemployment Survey (2011-12) , Para no 1.8.11.1 page no A-12)

This deliberate exclusion of domestic work related to “processing of primary products for own consumption” from the scope of “economic activity” is justified from the perspective of estimation of real employment opportunities available to rural female, particularly those are “poor” and outside any formal educational system. By giving such females, the stamp of “worker” is a meaningless glorification towards the satisfaction of policy makers. In this connection, the following quote from Dantawala committee report of 1970, which provided the framework for NSS Employment-Unemployment survey, is still relevant for the Indian scenario of female participation in labour force:

On the basis of our present knowledge, it is difficult to predict with any accuracy either direction of the change in the labour participation rate or the kind of transformation that may take place in the pattern of labour supply. All one can say is that it would be a serious mistake to proceed on the assumption that the entire labour force is of a homogenous character and make estimates of increases in the labour force simply on the basis of population projections and aggregate sex-age specific participation in data relating to the past. (Planning Commission (1970) page 20, para 3.30)

It is nobody’s case that there is no shortcoming of the methodology and implementation practice of NSSO surveys. Issues that are being raised by statistical community itself about these shortcomings are significant and substantial. Since NSSO surveys deal with the subjects – poverty, unemployment, malnutrition etc.- that are considered subjects of utmost important by policy makers, it is natural that such shortcomings will be looked with a glass politically tinted, and one need not be squeamish about that. Angus Deaton and Valerie Kozel, while discussing the utility of NSSO survey data for estimation of poverty in India provided the most balanced view about these surveys.

“Inevitably, mistakes will be made, surveys will be compromised by internal or external factors, so that poverty assessments will have to be made using imperfectly comparable surveys. The Indian experience illustrates the possibility of repair to enhance the credibility of estimates. But that experience also made it clear that repairs, however creative, are a poor substitute for the collection of clean, credible, and comprehensive data. What are convincing assumptions to one can be absurd to another, and people’s political positions seem to play a role in the assumptions that they are prepared to make. Nevertheless, the Indian debate has shown that discussion and advance is possible, even among those with very different preconceptions, and that the balance of opinion can be changed by well-reasoned and transparent argument (Deaton & Kozel 2004 page 43)”

Keeping the issue of definition of “work/employment” aside, the authors’ computation makes one point crystal clear- rising female full-time participation in education would be one possible factor for declining FLFPR in rural areas.

Indicator 3: Decline in Life Expectancy at Birth: An Untenable Narrative

The Human Development Index (HDI) has been introduced by UN to provide an aggregate measure of human development of its member countries. HDI is computed as a geometric mean of 3 constituent indices reflecting 3 dimensions of human development, namely, a long and healthy life, access to knowledge and a decent standard of living. The HDI is the geometric mean of normalized indices for each of these three dimensions.  The proxy for “a long and healthy life” is the indicator “Life Expectancy at Birth” which accounts for 35% of the overall HDI (UNDP 2021/22)

Life Expectancy at any given age (age is 0 for a newborn) is the residual years that a person is expected to live. The HDI uses, “Life Expectancy at Birth”. There are two variants of this measure- cohort life expectancy and period life expectancy.  

For the first variety, one identifies a group of people born in a particular year and track their deaths over time. It is then simple to work out the average number of years lived by that group. Since it is not possible to estimate the cohort life expectancy in this way unless all members of the cohort have died, a statistical estimate is worked out by combining the past mortality rates of the cohort at a given point of time and a projected mortality rate for the future.   

For the “period life expectancy” measure, one first estimates mortality rate of the current period (year) for a particular group of people defined by any attribute (say age group by sex) and then assumes that rate to remain constant over time. Under this assumption, it would be possible to work out average life expectancy for that group for that specific year. Since mortality rate of a given population changes over time, this measure of Life Expectancy would also change. More importantly, one needs to estimate mortality rates based on the deaths that have occurred in a particular year and so it would change if there were a spike in death rates, say due to a pandemic, in a particular year. It appears that authors of the working paper of EAC-PM have overlooked this issue. As soon as death rates fall in subsequent years, we will again see changes in opposite direction for that year. The published figures for 2023 by WHO confirm this.

Table 3. 1 Year wise Life Expectancy for India

Year	Both sexes	Female	Male
2010	66.9	68.6	65.3
2011	67.4	69.1	65.8
2012	67.9	69.6	66.3
2013	68.5	70.1	67
2014	69.1	70.6	67.7
2015	69.6	71.1	68.3
2016	70.1	71.5	68.8
2017	70.5	71.9	69.2
2018	70.7	72.1	69.4
2019	70.9	72.4	69.5
2020	70.1	71.8	68.6
2021	67.2	68.9	65.8
2022	67.7	69.4	66.3
2023	72	73.6	70.5

Once we understand that the estimated decline in Life Expectancy at Birth for India does not suffer from any conceptual flaws per se, we need to examine the computational aspect of mortality rates for different segments of the Indian population that WHO have used for the pandemic years. This issue has attracted sharp criticism from the authors of WP-1 because of its impact on estimation of life expectancy. We must emphasise here that WHO has not undertaken any new adjustment to the officially reported All Cause Mortality (ACM) for Indian population. What has been revised is the officially declared mortality due to COVID-19. Estimation of Life Expectancy during the pandemic years critically depends on a correct estimate of death due to  COVID-19 for the simple reason that such deaths were much higher in the adult age-groups. Any spike in the mortality of higher age groups would then bring down the life expectancy at birth. Bhattacharya et al has pointed out how social stigma associated with segregation of COVID-19 patients led to a situation of “creating a fear among the public and is acting as a deterrent to the effective management of the disease, particularly in the urban setup (Bhattacharya Prama et al 2020).” So, there is a distinct possibility of under reporting of COVID related deaths.

Since COVID-19 is a new phenomenon, death due to COVID-19 would only add to number of deaths that would have happened in the absence of COVID-19. WHO has defined this excess death as “the mortality above what would be expected based on the non-crisis mortality rate in the population of interest”.

WHO has identified two countries (India and Indonesia) for which ACM data is available only at sub-regional level and not at national level. For these two countries, WHO constructed “a multinomial model, based on the assumption that the fractions of deaths in sub-regions remain approximately constant over time.” (Knutson Victoria et al 2023). Excess death estimated by WHO’s method has been compared with the results of few other methods and differences are within reasonable bounds.

Table 3.2 Estimates of Excess Death due to COVID-19 by different approaches

Approach	Estimate (10^6)	95% Confidence interval	Period
Naïve	5.04	(4.48,5.59)	Jan20-Dec21
WHO	4.74	(3.31,6.48)	Jan20-Dec21
The Economist	4.86	(1.70, 8.47)	Jan20-Dec21
IHME	4.07	(3.71, 4.36)	Jan20-Dec21
Naïve	4.29	(4.00,4.59)	June 20-June21
WHO	4.33	(2.85,6.13)	June 20-June21
Jha et. al.	3.23	(3.06, 3.39)	June 20-June21
Naïve	3.96	(3.62, 4.29)	April20-June21
WHO	3.99	(2.40, 5.95)	April20-June21
Anand et.al. 2021 Method 1	3.4		April20-June21
Anand et.al. 2021 Method 2	4		April20-June21
Anand et.al. 2021 Method 3	4.9		April20-June21

Note:     The estimate by Jha et al. (2022) is for excess COVID-19 deaths. The naive estimates are based on the ACM estimates  where Y_t,1 is the observed ACM from the available states, and s the estimated fraction of deaths available in month t. The estimate by Jha et al.  is based on a nationally representative telephone survey, a government survey that covers 0.14 million adults and the Government of India’s data from facility-based deaths and CRS deaths in 10 states. Anand et al. (2021) use three methods: Indian States’ CRS (method 1), international age-specific infection fatality rates applied to Indian demography (method 2) and seroprevalence and a household survey (method 3) (quoted from Knutson Victoria 2023 ).

A few researchers have dealt with this issue of estimating death rates directly attributable to COVID-19 using micro level sample data. I would like to refer two studies for their innovative use of covid related death rates of well demarcated community of people to work out a national level estimate of death due to COVID-19. The first study was by Christophe Z. Guilmoto (2022) and the second one is by Acosta et. al. (2022) . Both the studies estimated COVID-19 related deaths at a much higher rate as compared to Indian official estimates. Some of the details of these two studies are given in end note2[i] . Although many journalistic observations exist about actual death count due to COVI-19 in India, it would be difficult to arrive at a robust and reliable estimate of the actual number. ( see articles in New York Times, BBC). Many assumptions are made by WHO and other researchers to arrive at an estimate of COVID-19 death in India, but these are bonafide assumptions and not to be colored with any ulterior motives.

To sum up, there is neither unquestionable evidence to accept the India’s Life Expectancy as estimated by WHO for the two pandemic years nor are there enough supporting materials to unequivocally accept the official ones. The fact that Life Expectancy estimation for India by WHO has resumed the pre-COVID trend after the two pandemic years of 2021 and 2022 should assure Indian policy makers about the unbiased approach of WHO. The suggestion by the authors of Reversing the Gaze, “that the Registrar General of India should timely publish life expectancy estimates every year” is highly recommended but not for the alleged unfounded observation that “one-sided adjustments and circular references are routinely done to India-related data by international agencies.”

see the footnote below

---

[i] Guilmoto identified four different samples of Indian populations, having complete data on their demographic structure and death occurrence. Selection of the four samples were largely subjective and guided by the availability of full data and meeting three criteria for sample selection. These were – reliability of death estimation, regional   representativeness, and demographic characteristics. The author’s estimate for correction factor to be applied to official estimate for the country was 7 and 8.6 for two samples.

Acosta et. al. (2022) have estimated excess mortality in the State of Gujarat, India, during the COVID-19 pandemic (March 2020-April 2021) by using data from civil death registers from a convenience sample of 90 municipalities across the state of Gujarat. This study found that “the largest increases in mortality occurred in the second wave of the pandemic, across all municipalities”. It was further found that “over 80% of the municipalities experienced increases in mortality over 100% in all demographic groups except the two youngest age cohorts. Lastly, in over 50% of the municipalities, males and the 40 to 65 years groups experienced an increase of over 500%.

end of footnote

Section 2: Analysis of Methodological criticisms of WP-2

This paper’s stated objective is to provide a “quantitative analysis of the data quality of the National Sample Survey (NSS) in terms of three estimates, (i) the proportion of the rural population, (ii) the proportion of the Scheduled Caste (SC) population, and (iii) the proportion of the working-age (age between 15 and 59 years) population”.

Although the above noted estimates are published in the reports of these surveys, these are neither the focus of any NSS survey nor are these used in any survey report from this perspective.  On the contrary, population census data is used for allocation of total sample to various stages of selection of survey households at village and urban block level[1]. To be consistent with survey design, it is necessary to estimate population from the selected households and work out various ratios like labour force participation rate, distribution of persons of each sector of each State/UT over 12 classes of MPCE (monthly per capita consumption expenditure).

Notwithstanding the above, drawing direct comparison between the estimated rural and urban population data obtained from this survey and the corresponding figures from the 2011 census data requires careful consideration due to several factors. Firstly, the census figures were collected as on a particular day- 31st March 2011, whereas this survey was conducted in 4 sub-round of 3 months each, spanning July 2011 to June 2012. The figures from the survey are not expected to match with that of census data. Secondly, there are homeless people, convicted prisoners in jail, floating populations who are not covered by NSS surveys. Such people would be more likely to be in urban areas and this would negatively impact share of urban population. Nevertheless, there is no straight forward answer to the impact that these exclusions would have on the target measures like rate of unemployment or incidence of poverty at sub-national levels.

Keeping the above caveat in perspective, the following paragraphs explain that the application of the new concept of Total Sample Error (TSE) is a completely misguided use of Meng’s formulation that have been specially developed for examining  sampling issues related to Big data and/or large non probabilistically obtained administrative data.

Professor Meng has clarified that his paper is focussed on population inferences from Big Data (Meng 2018). Dr. Ravi and her colleagues have used the formula given in the Meng’s paper to compute sampling bias- the difference between the sample average of a variable of interest and its population average . This has been worked out, in Meng’s paper, as a product of three terms: (1) a data quality measure, (2) a data quantity measure, and (3) a problem difficulty measure. In case of NSS survey, the data quality is to be ascertained at the final stage sampling process- from a village or an urban block because final estimates are built up from that level. There is no evidence so far that NSS surveys suffer from a data quality issue because respondents are non-cooperative or deliberately providing wrong numbers. Furthermore, implementation of interpenetrating subsample and use of two sets of investigators (central sampling and state sampling) would clearly bring out any abnormal variation in estimation of target variables. In fact, Dr. Meng agrees that “under a genuine probabilistic sampling, the chance that a particular value of G (i.e., study variable, my addition) is recorded/reported or not should not depend on the value itself. Consequently, ρR,G should be zero on average.” So would be the sampling bias.

In another paper on this subject, the same inference has been drawn for a probabilistic sample survey, – “random sampling or any sample design that provides constant inclusions probabilities, the term for data quality ER(𝜌RX) is 0 and thus the bias is also 0 (Biemer and Amaya 2018)

Unfortunately, the authors of WP-2 have mechanically used Dr. Meng’s approach developed for Big data/Administrative data to assess quality of NSS survey data, without critically reviewing the NSS sampling design. The way to have an independent assessment of any sampling estimate was nicely brought out with an example by none other than founding fathers of NSS – Prof P. C. Mahalonobis and D. B. Lahiri. In their paper (1961) “Analysis of Errors in Censuses and Surveys with Special Reference to Experience in India” they compared the “results obtained on the basis of complete enumeration and sample survey against a third, but extremely reliable, figure. This could be carried out for two consecutive years – 1944-45 and 1945-46- for Jute Crop of Bengal” ( emphasis added and see end note 3 for details[i]).

Nevertheless, the author’s use of Meng’s approach to measuring of Total Sampling Error has inadvertently brought into focus the need for using “found data” in conjunction with the usual probabilistically sampled data for estimation of many social and economic indicators of a country as large as India. Administrative data are a type of found data which gets routinely generated as a part of the official activities like getting a subsidy, registering of birth and death, registering of vehicles, getting unemployment benefit, income tax data etc.   Many official statistical authorities in advanced countries are already exploring big data or found data integration with survey data to achieve better quality of statistical estimates of important economic and social indicators. For example, Statistics New Zealand has established an Integrated Data Infrastructure (IDI) , sourcing data from government agencies, Stats NZ surveys, and non-government organisations (NGOs). Statistics Netherlands has established a Center for Big Data Statistics in 2016 “with the primary purpose of leveraging new and existing (big) data sources and techniques in order to arrive at better information about social themes such as the labour market, mobility, health, the energy transition and smart farming.” (Unique collaboration for big data research (cbs.nl) ). Statistics Canada uses administrative data to complement or to replace survey data.

Finally, it would be amiss if Dr. Ravi’s stringent criticism of NSS sampling design in her article “The sample is wrong” published in the Indian Express 7th July edition. Three areas of her concerns are: availability of data, transparent and robust statistical analysis, and data quality.

The first area of concern- more frequent and timely availability of survey data to policy makers for evaluating any policy intervention at national level- is indeed a crucial one, and it requires a coordinated effort between statistical authorities and government decision-makers to address it effectively. As a member of the EAC-PM the author can play a significant role in advocating for the importance of timely and frequent survey data.

As pointed out in the discussion on WP-2 above, the data quality measure used therein is meant for non-probabilistic data or for an integrated dataset of administrative data and a probabilistic sampling data. Thus, the author has not given any statistically valid argument to declare that “Sample is wrong”.

Dr. Ravi’s criticism about the quality of Indian Official Statistics, however methodologically flawed it might be, needs deep appreciation because it has highlighted the absence of any innovation in statistical methodology used in production of the critical social and economic indicators. in the last two decades. Thus, to paraphrase Agatha Christie, the Indian Statistical Mirror as shown to us by the official statistical system might not have “crack’d from side to side”, but it is gradually becoming covered with dust of obsolescence.

---

[1] NSSO follows a stratified multi-stage design for its surveys. The first stage units (FSU) are the 2001 census villages (Panchayat wards in case of Kerala) in the rural sector and Urban Frame Survey (UFS) blocks in the urban sector. For the rural sector, the list of 2001 census villages constitutes the sampling frame. For the urban sector, the list of UFS blocks (2007-12) is considered as the sampling frame

---

[i] Jute being a cash crop of international importance, accurate export trade figures were available after a gap of 15 months of harvest. So, there was a need for a quick estimation of production. The official approach to meet this demand was to carry out a plot-to-plot enumeration. Indian Statistical Institute undertook two sample surveys- one for estimation of acreages and subsequently another for post-harvest yields. Surveys were conducted using two interpenetrating sub-samples (IPNS) which were covered by different parties of investigators. These two independently computed values were compared with the trade figures. The result was follows.

” in both the years the official forecasts based on complete count were both very much out whereas the sample survey estimates were quite close to the trade figures. The two sub-sample (IPNS) estimates in both years agreed with the trade estimates within roughly 3 per cent while the estimates based on the so-called complete count differed from the trade figures by 27.2 per cent in 1944-45 and 16.6 per cent in 1945-46” (Mahalanobis  and Lahiri (1961) page 328-329.

Quoted References:Acosta Rolando J , Biraj Patnaik ,and others, All-cause excess mortality in the State of Gujarat, India, during the COVID-19 pandemic (March 2020-April 2021) medRxiv preprint January 19, 2022

https://doi.org/10.1101/2021.08.22.21262432

Bhandari N, Bahl R, Taneja S, de Onis M, Bhan MK. Growth performance of affluent Indian children is similar to that in developed countries. Bull WHO 2002;80: 189–95

Bhandari Nita, Sunita Taneja,and others, Implementation of the WHO Multicentre Growth Reference Study in India Food and Nutrition Bulletin, vol. 25, no. 1 (supplement 1) 2004, The United Nations University

Bhattacharya Prama, Debanjan Banerjee, TS Sathyanarayana Rao, The “Untold” Side of COVID-19: Social Stigma and Its Consequences in India  , Indian Journal of Psychological Medicine | Volume 42 | Issue 4 | July 2020

Biemer Paul P. and Ashley Amaya, Total Error Frameworks for Found Data in Big Data Meets Survey Science: A Collection of Innovative Methods, First Edition. Edited by Craig A. Hill and Others, 2021

Childs, Jennifer. H., Fobia, Aleia C., King, R. and Morales, G. (2019). Trust and Credibility in the U.S. Federal Statistical System. Survey Methods: Insights from the Field. Survey Insights: Methods from the Field. Retrieved from https://surveyinsights.org/?p=10663

Coase Ronald (1994) (1994) , “How should economists choose?”, page 27 in the book  “Essays in economics and economists” University of Chicago Press

Das Sonali, Sonali Jain-Chandra, Kalpana Kochhar, and Naresh Kumar , Women Workers in India: Why So Few Among So Many? IMF Working Paper, March 2015

Deaton Angus and Valerie Kozel, Data and dogma: the great Indian poverty debate,  World Bank, September 2004

Debroy Bibek, The Janus of India’s official statistics, The New Indian Express, June 17, 2023

de Onis Mercedes, Cutberto Garza, Cesar G. Victora, Adelheid W. Onyango, Edward A. Frongillo, and Jose Martines, The WHO Multicentre Growth Reference Study: Planning, study design, and methodology   Food and Nutrition Bulletin, vol. 25, no. 1 (supplement 1) 2004, The United Nations University

de Onis Mercedes, Adelheid Onyango, and Others, Worldwide implementation of the WHO Child Growth Standards, Public Health Nutrition: 15(9), 1603–1610 April 2012

De Onis Mercedes  as Coordinator of (WHO Multicentre Growth Reference Study Group) , Assessment of differences in linear growth among populations in the WHO Multicentre Growth Reference Study, Acta Pædiatrica, Suppl 450, page 56-65, 2006.

Gundimeda Haripriya , Sanjeev Sanyal & others , 2007 Estimating the value of educational capital formation in India, ,TERI Press New Delhi 2006

Hermanussen, M., Stec, K., Aßmann, C., Meigen, C., & Van Buuren, S. (2015). Synthetic growth reference charts. American Journal of Human Biology, 28(1), 98–111

Guilmoto CZ (2022) An alternative estimation of the death toll of the Covid-19 pandemic in India. PLoS ONE 17(2): e0263187.

Kapsos Steven, Evangelia Bourmpoula, and Andrea Silberman   Why is female labour force participation declining so sharply in India : ILO Research Paper No. 10 July 2014 ,International Labour Office

Khadilkar V, Khadilkar AV, Kajale N. Indian growth references from 0‑18‑year‑old children and adolescents ‑ A comparison of two methods. Indian J Endocr Metab 2019;23:635-44

Khadilkar V, Ekbote V, Gondhalekar K, Khadilkar A. Comparison of nutritional status of under-five indian children (NFHS 4 Data) using WHO 2006 charts and 2019 Indian synthetic charts. Indian J Endocr Metab 2021;25:136-41

Knutson Victoria, Serge Aleshin- Guedel, and others, Estimating global and Country-Specific Excess Mortality during COVID-19 Pandemic, Ann. Appl. Stat. 17(2): 1353-1374, June 2023

Laurence M. Grummer-Strawn, Chris Reinold and Nancy F. Krebs Use of World Health Organization and CDC Growth Charts for Children Aged 0-59 Months in the United States Author(s): Source: Morbidity and Mortality Weekly Report: Recommendations and Reports, Vol. 59, No. RR -9 (September 10, 2010), pp. 1-14

Mahalanobis P. C. and D. B. Lahiri, Analysis of errors in censuses and surveys with special reference to experience in India, Sankhyā: The Indian Journal of Statistics, Series A (1961-2002), Vol. 23, No. 4 (Nov. 1961), pp. 325-358

Mehta S, Oza C, Karguppikar M, Khadilkar V, Khadilkar A. Field testing of synthetic growth charts in 1–60‑month‑old Indian children. Indian J Endocr Metab 2022;26:180-5

Meng  Xiao-Li , statistical paradises and paradoxes in Big Data (i): Law of Large Populations, Big Data Paradox, and the 2016 US Presidential Election, The Annals of Applied Statistics 2018, Vol. 12, No. 2, 685–726

NSS Report No. 554 (68/10/1): Employment and Unemployment Situation in India, 2011-12 Para 2.10.1 Page 12 January 2014

NSS 68th Round Employment-Unemployment Survey (2011-12)- Instructions to Field Staff, Vol. I

Planning Commission, Government of India (1970), Report of the Committee of Experts on Unemployment Estimates. ( also known as Dantwala committee report)

Ravi Shamika, Rahul Ahluwalia,and Sofi Bergkvist,   Health and Morbidity in India (2004-2014), Brookings India Research Paper, New Delhi  2016

Reddy D. Narasimha , Female Work Participation in India: Facts, Problems, and Policies, Indian Journal of Industrial Relations, Vol. 15, No. 2 (Oct., 1979), pp. 197-212

Sinha J. N., Dynamics of female participation in economic activity in a developing economy in Proceedings of the World Population. Conference, 1965 Volume IV: UN New York 1967

UNDP, Technical Notes: Human Development Report 2021/22

hdr.undp.org/system/files/documents/technical-notes-calculating-human-development-indices.pdf

United Nations, Resolution adopted by the Economic and Social Council on 24 July 2013 E/RES/2013/21

World Health Organization (2023) Methods for estimating the excess mortality associated with the COVID-19 pandemic World Health Organisation

Other References

Bardhan Pranab, The State of Indian Economic Statistics: Data Quantity and Quality Issues, University of California at Berkeley

Bhattacharya Pramit, India’s Statistical System: Past, Present, Future , Carnegie Endowment for International Peace, 2023

Biemer Paul P., Edith de Leeuw and Others (Editors), Total Survey Error in Practice  2017 John Wiley & Sons

de Onis Mercedes and Francesco Branca, Childhood stunting: a global perspective, Maternal & Child Nutrition (2016), 12 (Suppl. 1), pp. 12–2

Instructions to Field Staff, Vol. I: NSS 68th Round Employment-Unemployment Survey; July 2011 – June 2012

Mehrotra, S., & Parida, J. K. Why is the Labour Force Participation of Women Declining in India? World Development 2017   http://dx.doi.org/10.1016/j.worlddev.2017.05.003

Xiao-Li Meng & Xianchao Xie (2014) I Got More Data, My Model is More Refined, but My Estimator is Getting Worse! Am I Just Dumb? Econometric Reviews, 33:1-4, 218-250,

Neff, Daniel; Sen, Kunal; Kling, Veronika (2012) : The Puzzling Decline in Rural Women’s Labor Force Participation in India: A Re-examination, GIGA Working Papers, No. 196

Prabhat Jha, Yashwant Deshmukh, and others,  COVID mortality in India: National survey data and health facility deaths Science, 375 (6581)

Rao Talluri, Official Statistics in India: The Past and the Present, Journal of Official Statistics 26(2):215-231 , June 2010

World Health Organization and the United Nations Children’s Fund (UNICEF), Recommendations for data collection, analysis and reporting on anthropometric indicators in children under 5 years old, 2019