“Data can’t speak for itself; it’s up to you to give it a voice. Try to speak truthfully.” – Ronald Coase

Introduction

Recently a debate took place on the pages of Indian Express (starting on July 7 edition) about the quality of survey-based data released by the National Sample Survey Organizations (NSSO). Before that doubts about quality of Indian official statistics data were aired through two working papers issued by Economic Advisory Council of the Prime Minister (EAC-PM). When a couple of eminent economists of impeccable credentials and having advisory role to none other than the honourable prime minster of India raises serious doubt about the quality and accuracy of some key statistical products that are supposed to reflect, directly or indirectly, the performance of the economy, it is incumbent on the statistical professionals to take these criticisms seriously and put out their non-partisan views to the public at large.

In 1994, the General Assembly of United Nations approved 10 Fundamental Principles of Official Statistics, as adopted by the United Nations Economic Commission for Europe (UNECE) in 1992. The first principle explains why official statistics is an “indispensable element in the information system of a democratic society”.  The second principle explicates the requirements to “retain trust in official statistics” – an ethical and strictly professional approach towards adoption of “methods and procedures for the collection, processing, storage and presentation of statistical data” (United Nations, Resolution adopted by the Economic and Social Council on 24 July 2013).

The OECD Committee on Statistics in 2008 initiated an effort to measure “Trust in Official Statistics” (OECD 2011) . In July 2012, the model questionnaire for measuring trust in official statistics was published in the Statistics Newsletter of OECD (OECD Statistics 2012). The approach adopted for this measurement was based on three factors- Structural, Statistical and Reputation. Impartiality (that is no political interference) and independence, amongst others, were identified as structural factors while compliance with international standards was considered as one of the statistical factors. Six national statistical agencies participated in testing of this questionnaire.

Using the OECD questionnaire as the base, 6 US statistical agencies (Federal Statistical System-FSS) developed a trust measuring questionnaire in the US context and used Gallup daily poll to add its questions between February 2012 and September 2013. One major conclusion of this survey was-

Data users have formed a relationship with statistics and the statistical institutions that produce this data; accordingly, we interpret their higher levels of trust to be a result of a relationship formed between the data user and the data.  (Childs, Jennifer er al, 2019)

Incidentally, both Dr. Sanjeev Sanyal and Dr. Shamika Ravi – lead authors of the two monographs under discussion -have used NSSO data in some of their earlier research articles. As users of Indian survey data, they did not express any reservation about data quality or underlying sampling methodology. They trusted NSSO data. The monograph titled Estimating the value of educational capital formation in India is based on data of surveys of employment and unemployment conducted for 1993 and 1998 by NSSO. The same is true for the article titled Health and Morbidity in India (2004-2014), authored by Shamika Ravi. It was published under the banner of Brookings India carries out a comparative study of healthcare in India over the ten years using data of round 60 and round 71 of NSSO. It may not be out of place to note that Sanyal’s article was written under the auspices of United Nations.  

The present article aims to examine the veracity of the criticisms raised by these two members of the EAC-PM regarding the quality of estimation of key policy performance indicators by NSSO surveys. However, prior to delving into the deficiencies of indicators highlighted by the economist members of EAC-PM, it is pertinent to acknowledge the valid concerns articulated by the EAC-PM’s chairman pertaining to the broader management issues of the official statistical system (Bibek Debroy, June 2023). Some of the well-known deficiencies flagged by the chairman are -inordinate delay in releasing of survey data, lack of proficient manpower, and outdated recruitment process of qualified statisticians.

The responsibility of rectifying these deficiencies and effecting necessary resolutions inherently falls within the purview of the government machinery. The chairman of the EAC-PM should wield the authority and agency to collaboratively seek remedies from the relevant governmental entities. Addressing these systemic concerns will not only enhance the statistical capabilities of the nation but also fortify the foundation upon which sound policy decisions can be formulated and assessed.

In the following two sections, I examine the maintainability of various statistical issues raised by the following two working papers of EAC-PM.

1. Reversing the Gaze:  Re-examining Estimates of India’s Development Indicators by International Organisation:  EAC-PM Working Paper Series EAC-PM/WP/14/2023; March 2023 Authors: Sanjeev Sanyal, Aakanksha Arora, and Srishti Chauhan; henceforth, referred to as WP1.
2. Assessing the National Surveys for its Representativeness:   An Analysis of the Data Quality of the National Sample Survey (NSS)- July 2023 Authors: Shamika Ravi, Mudit Kapoor, and S V Subramanian; henceforth, referred to as WP 2

Section I:  Criticisms of methodologies adopted for computation of different indicators in WP 1

This working paper deals with accuracy and soundness of “three widely used data-driven development indicators” which according to the authors “seem to stagnate or even deteriorate despite a rise in per capita income in India” (Executive summary pp3). These three indicators are             ” Childhood Stunting”, “Female Labour Force Participation Rate (FLFPR)”, and “Life Expectancy at Birth”.

Indicator No-1 Stunting

Childhood malnutrition has deleterious effects on a child’s growth over age. Two important measures of these effects are wasting and stunting. Wasting is defined as low weight-for-height. Stunting is defined as low height-for-age. The prevalence of stunting amongst Indian children in the age group of 0-5 years has been declining over the years. The authors have quoted the following trend- 35.5% in 2019-21(rural 37.3%; urban 30.1%)- down from 38.4% In 2015-16 and 48% in 2005-06. The source of this data is National Family Health Survey (NFHS). The estimates of stunting have been worked out using the WHO’s global standard for this measure. The authors have questioned the methodology adopted by WHO in deriving this standard and, furthermore, its applicability in the Indian context.

Who Standard for Stunting

WHO conducted a multicentre growth reference study (MGRS) between 1997 and 2003 in 6 countries with the objective of deriving a “single international reference representing the best standard possible of optimal growth for all children < 5 y of age” ( de Onis et al 2004). The selected countries were Brazil, Ghana, India, Norway, Oman and the United States, covering all geographical regions.

The important point to note is the word “optimal growth”. The purpose of the MGRS study is to arrive at a global standard for the growth in height of a healthy and well-fed child. The study is not aimed to prescribe a global standard for height for children in the selected age group. To be more precise, WHO standard is with respect to “Growth velocity” of “height-for-age”. It is defined as the change in height or length over time.

Since the objective of the MRGS study is to find “optimal growth” of breast-fed healthy children, surveys were conducted to “identify socioeconomic characteristics that could be used to select groups whose growth was not environmentally constrained”. Thus, “Families’ low socioeconomic status was the most common reason for ineligibility in Brazil, Ghana, India and Oman”.   In India, the “sample was drawn from 58 affluent neighbourhoods in South Delhi. This community was selected to facilitate the recruitment of children who had at least one parent with 17 or more years of education, a key factor associated with unconstrained child growth in this setting. (Bhandari Nita et al 2004)” Before New Delhi was chosen for the MRGS study, a survey was conducted in south Delhi in order to “determine whether affluent population in south Delhi had a growth performance similar to that in developed countries” (Bhandari Nita et al 2002). The survey found that the growth performance of children of the affluent population in Delhi were “close to the NCHS/WHO reference population with regard to anthropometric indicators”. Even better growth was registered for children of parents with higher educational background. 

The issue of using a global standard for estimating prevalence of children malnutrition has been criticised by a relatively small group of researchers. Since mean height and weight of children at various age group vary from country to country, some experts argued that it is expected that required growth curve should be country specific. So, a new growth curve, called synthetic growth curve has been proposed and applied to a small group of countries like Indonesia, Malwai and a few more. There is a confusion about the use of CDC growth curve in USA as against WHO growth curve. But CDC has clarified the purpose of introducing CDC growth reference curve:

 “Whereas the WHO charts describe growth of healthy children in optimal conditions, the 2000 CDC growth charts are a growth reference, not a standard, and describe how certain children grew in a particular place and time. The CDC charts describe the growth of children in the United States during a span of approximately 30 years (1963-1994) (Grummer-Strawn et al 2010).”

The idea of synthetic growth curve was proposed by Michael Hermanussen and others in a 2015 paper. Based on “a reference combination of longitudinal and cross-sectional modern and historic growth studies with data on height and weight, global mean values and a limited number of Principal Components that characterizes the variability of growth in the reference combination” were calculated (Hermanussen Michael et al 2015). It is surprising that there was no transformation of the underlying variables for bringing normality so that 2-standard deviation outlier detection would be more appropriate. In fact, the reported empirical results based on data of 196 females and 197 males, collected from 2000(!) earlier studies cannot inspire confidence in the robustness of this methodology.

In the Indian context few researchers have sought to estimate prevalence of stunting using India specific reference growth curve using this methodology. All of them used methodology proposed by Hermanussen and his collaborators. Khadilkar et al (2019) used 46,421 children’s data for computing reference synthetic growth chart. To produce synthetic references, arithmetic means of height and weight at key ages viz. birth, 2, 6, 12 and 15 years for girls and birth, 2, 6, 14 and 18 years for boys were used. Obviously synthetic references were not for “optimal” growth but for average growth of selected age-groups. To compare this with WHO standard is wrong to begin with. Since the synthetic reference chart for India was not developed as the desired or “optimal growth” chart based on carefully selected reference group of children of India as MRGS study did, the comparison that was carried out using that synthetic chart is not a valid comparison (Khadilkar 2025).  

In another study of comparison of the results of prevalence of stunting as between the one using WHO standard with the India specific synthetic refence chart, the researchers selected its sample units in the following way:

“Apparently healthy children between the ages of 0 and 60 months attending an urban health centre and a rural primary health centre for vaccination were included in this study” (Sujit Mehta et al 2022). The words “apparently healthy children” used for sampling method belies the quality of such samples for the intended comparison.

WHO conducted a survey of the status of implementation of “WHO Child Growth Standards” by its member countries (de Onis 2012). Out of 219 countries contacted for the survey, responses were received from 180 countries, of which 125 informed adoptions of WHO standards. Non-response was observed mostly for small countries. India has reported adoption of WHO standards in compilation of the indicators for child malnutrition. It is necessary to emphasize that non-adoption of international standards on computation of key national performance indicators like GDP, Export/Import data, compilation of balance of payment data cannot be an option for a country that wants to become a global superpower.

Finally, a comparative position of India among some selected countries in terms of this indicator on stunting, per the WHO standard, should be of greater importance to policy makers. The data given in the following table speaks for itself.

Table 1.1: Trend in prevalence of stunting among children under age 5 for selected countries.

	Prevalence of stunting, height for age (% of children under 5)
Country	Starting year	Data for starting year	End Year	Data for end Year	Number of years between two data points	Average Rate of  Yearly Reduction
Bangladesh	2000	51.1	2019	28	19	-3.2%
Brazil	1996	13	2007	7	11	-5.6%
China	2000	17.8	2017	4.8	17	-7.7%
India	1999	54.2	2017	34.7	18	-2.5%
Indonesia	2000	42.4	2018	30.8	18	-1.8%
Malaysia	1999	20.7	2019	21.8	20	0.3%
Pakistan	2001	41.4	2018	37.6	17	-0.6%
South Africa	1999	30.1	2017	21.4	18	-1.9%
Sri Lanka	2000	18.3	2016	17.3	16	-0.4%
Vietnam	2000	43.2	2020	19.6	20	-4.0%

Note to the table 1.1: Start year is the nearest year from 2000 for which data is available. End year is that year for which data is available and is closest to 2022. Average yearly reduction is computed by log of ratio of start data to end data divided by total number of elapsed years between these two years. Source: https://ourworldindata.org/grapher/share-of-children-younger-than-5-who-suffer-from-stunting

Indicator No 2. Female Labour Force Participation Rate (FLFPR)

The second section of WP1 is titled as “Flaws in Female Labour Force Participation Estimation: ILO’s Unworkable Maths”. The content of the section thereafter “reverses its gaze”, as reflected in the following line in the first para of the section itself:

“this section finds quite the different that how not following an appropriate internationally accepted standard leads to serious underestimation of a relevant labour market and gender equality statistic i.e., the Female Labour Force Participation Rate for India.”

Obviously, the authors are confused about their target of criticism- ILO or Indian estimates of the indicator.  The source of this confusion lies in the fact that declining rural FLFPR of India is a much analysed and debated subject.

In the World Population Conference held at Belgrade in 1965, J.N.Sinha of India, suggested a U-type movement in FLFPR as per capita income increases “The rate’s decline with an increase in income up to $500, but begin to rise with further gains in income.  ..  Within the rural areas, the rates vary inversely with agricultural prosperity and the proportion of non-agricultural work force. Labour force participation of women also declines with literacy, but female education above the matriculation level favours higher rates of employment (Sinha 1965).[1]”

In 2011, Bhalla and Kaur wrote a working paper titled Labour Force Participation of Women in India: Some facts, some queries. They made one interesting extension to the computation of labour force by adding students in the age group 15-59 to the labour force. They called it adjusted labour force and computed the FLFPR based on NSSO defined labour force as well as the” adjusted labour force”

---

[1] See also D. Narasimha Reddy, Female Work Participation in India: Facts, Problems, and Policies Source: Indian Journal of Industrial Relations, Vol. 15, No. 2 (Oct., 1979)

Table 2.1  Percentage of female population in Labour Force and adjusted labour Force

Area	Year->		1983	1993/94	1999/00	2004/05	2007/08
Rural	Unadjusted Labour Force		45.1	53.1	45.2	44.7	37.6
	Adjusted Labour Force		46.8	55.9	49.5	50.4	44.4
Urban	Unadjusted Labour Force		23	23	22.5	24.3	19.7
	Adjusted Labour Force		30.5	33.3	33.4	35.5	32.3

In a 2014 ILO research paper, Steven Kapsos and his two fellow ILO economists expanded the definitional boundary of labour force used by NSSO by including those “people who were engaged in domestic economically gainful activities such as collection of firewood, poultry etc.” ( Kapsos 2014). Based on this expanded size of labour force, they computed a new LFPR, termed as Augmented LFPR. The following table gives their computation of Augmented LFPR for both sexes and regions.

Table 2.2: Labour force participation rates (%), UPSS and augmented definition, by sex and area

	UPSS Definition					Augmented definition
Area	1994	2000	2005	2010	2012	1994	2000	2005	2010	2012
Rural	49	45.4	49.4	37.8	35.8	80.8	77	76.3	70.1	66.8
Urban	23.8	20.8	24.4	19.4	20.5	45.2	38.5	39.1	35.9	32.1
All areas	42.7	38.9	42.7	32.6	31.2	71.8	66.8	66.4	60.3	56.4

UPSS: usual principal and subsidiary status; Age group is 15+.

The expanded definition of labour force did not result in any substantial difference in the trend observed in the NSSO defined rural female labour force participation rate. This result of augmentation is not unexpected given the substantial share of persons engaged in domestic activities in rural India- see Table 2.3 below.

Table 2.3: Shares of the working-age population engaged in domestic duties.  

Gender	Region	1994	2000	2005	2010	2012
Female	Rural	42.4	43.9	39.8	49.4	49.9
Female	Urban	60.7	61.9	59.3	62.1	61.1
Male	Rural	0.4	0.4	0.4	0.5	0.5
Male	Urban	0.5	0.4	0.4	0.6	0.3

The ILO economists identified 4 key factors for changes in female participation rates- “increased attendance in education”, “increased household consumption levels”, changes in measurement of economic activities”, and “changes in employment opportunities” and carried out a statistical exercise to estimate the relative contribution of these factors towards the decline of FLFPR in rural areas. Their conclusion at the end is worth reproducing.

“The econometric results indicate that religion and social perceptions of women, women’s level of

education, household size and income, and the presence of young children in the household all influence the likelihood of India’s women to participate in the labour market. We find that structural

characteristics in the labour market have played a more important role than changes in the underlying characteristics of the female working-age population in influencing participation rates. These structural barriers, such as norms that inhibit women’s labour market options, in conjunction with a consistent decline in agricultural employment, are likely to be key factors in explaining the long-term stagnation in female participation rates.” (Kapsos 2014 ,page 31)

In a 2015 IMF Working paper, Sonal das & others carried out a regression analysis to identify determinants of female labor force participation in India in both urban and rural areas(Das Sonali 2015). Amongst the 5 “stylized facts” noted by the authors based on 5 NSSO EUS between 1993-94 and 2011-12, two are about causal factors for the observed declining trend in rural FLFPR:

• There is a U-shaped relationship between education and labor force participation rates of women

• Income has a dampening effect on female labor force participation rates

It is now beyond doubt that the issue of declining FLFPR in rural India is a known and well researched issue. Everyone who has dealt with the issue were aware of the differences in definitional boundary of “economic activity” adopted by NSSO surveys as compared to that of national income accounting.  The difference is spelt out in the 68^th round( NSS.

Although production of any good for own consumption is considered as economic activity by UN System of National Accounts, production of only primary goods for own consumption was considered as economic activity by NSSO for the purpose of the survey. While the former considers activities like own account processing of primary products as economic activities, processing of primary products for own consumption was not considered as economic activity in the NSS surveys.

This deliberate exclusion of domestic work related to “processing of primary products for own consumption” from the scope of “economic activity” is justified from the perspective of estimation of real employment opportunities available to rural female, particularly those are “poor” and outside any formal educational system. By giving such females the stamp of “worker” is a meaningless glorification towards the satisfaction of policy makers. In this connection, the following quote from Dantawala committee report of 1970, which provided the framework for NSS Employment-Unemployment survey, is still relevant for the Indian scenario of female participation in labour force:

On the basis of our present knowledge, it is difficult to predict with any accuracy either direction of the change in the labour participation rate or the kind of transformation that may take place in the pattern of labour supply. All one can say is that it would be a serious mistake to proceed on the assumption that the entire labour force is of a homogenous character and make estimates of increases in the labour force simply on the basis of population projections and aggregate sex-age specific participation in data relating to the past.

It is nobody’s case that there is no shortcoming of the methodology and implementation  practice of NSSO surveys. Issues that are being raised by statistical community itself about these shortcomings are significant and substantial. Since NSSO surveys deal with the subjects that are inherently political – poverty, unemployment, malnutrition etc. -it is natural that such shortcomings will be looked with a glass politically tinted, and one need not be squeamish about that. Angus Deaton and Valerie Kozel, while discussing the utility of NSSO survey data for estimation of poverty in India provided the most balanced view about these surveys.

“Inevitably, mistakes will be made, surveys will be compromised by internal or external factors, so that poverty assessments will have to be made using imperfectly comparable surveys. The Indian experience illustrates the possibility of repair to enhance the credibility of estimates. But that experience also made it clear that repairs, however creative, are a poor substitute for the collection of clean, credible, and comprehensive data. What are convincing assumptions to one can be absurd to another, and people’s political positions seem to play a role in the assumptions that they are prepared to make. Nevertheless, the Indian debate has shown that discussion and advance is possible, even among those with very different preconceptions, and that the balance of opinion can be changed by well-reasoned and transparent argument (Deaton & Kozel 2004)”

Keeping the issue of definition of “work/employment” aside, the authors’ computation makes one point crystal clear- rising female full-time participation in education would be one possible factor for declining FLFPR in rural areas. It must be noted here that higher FLFPT in rural areas as compared to in urban areas is a sign of higher incidence of poverty in rural areas.

Indicator 3: Decline in Life Expectancy at Birth: An Untenable Narrative

The Human Development Index (HDI) has been introduced by UN to provide an aggregate measure of human development of its member countries. HDI is computed as a geometric mean of 3 constituent indices reflecting 3 dimensions of human development, namely, a long and healthy life, access to knowledge and a decent standard of living. The HDI is the geometric mean of normalized indices for each of these three dimensions. The proxy for “a long and healthy life” is the indicator “Life Expectancy at Birth” which accounts for 35% of the overall HDI[1].

---

[1] Technical Notes: Human Development Report 2021/22

hdr.undp.org/system/files/documents/technical-notes-calculating-human-development-indices.pdf

Life Expectancy at any given age (age is 0 for a newborn) is the expected residual years that a person is expected to live. For HDI, “Life Expectancy at Birth” is needed. There are two variants of this measure- cohort life expectancy and period life expectancy

For the first variety, one identifies a group of people born in a particular year and track their deaths over time. It is then simple to work out the average number of years lived by that group. Since it is not possible to estimate the cohort life expectancy in this way unless all members of the cohort have died, a statistical estimate is worked out by combining the past mortality rates of the cohort at a given point of time and a projected mortality rate for the future.   

For “period life expectancy” we first estimate mortality rate of the current period (year) for a particular group of people defined by any attribute (say age group by sex) and then assume that rate to remain constant over time. Under this assumption, it would be possible to work out average life expectancy for that group for that specific year. Since mortality rate of a given population changes over time, this measure of Life Expectancy would also change. More importantly, we need to estimate mortality rates based on the deaths that have occurred in a particular year and so it would change if there is a spike in death rates, say due to a pandemic, in a particular year. It appears that authors of the working paper of EAC-PM have overlooked this issue. As soon as death rates fall in subsequent years, we will again see changes in opposite direction for that year. The published figures for 2023 by WHO confirms this.

Year	Both sexes	Female	Male
2010	66.9	68.6	65.3
2011	67.4	69.1	65.8
2012	67.9	69.6	66.3
2013	68.5	70.1	67
2014	69.1	70.6	67.7
2015	69.6	71.1	68.3
2016	70.1	71.5	68.8
2017	70.5	71.9	69.2
2018	70.7	72.1	69.4
2019	70.9	72.4	69.5
2020	70.1	71.8	68.6
2021	67.2	68.9	65.8
2022	67.7	69.4	66.3
2023	72	73.6	70.5

Once we understand that the estimated decline in Life Expectancy at Birth for India does not suffer from any conceptual flaws per se, we need to examine the computational aspect of mortality rates for different segments of the Indian population that WHO have used for the pandemic years. This issue has attracted sharp criticism from the authors of the paper under discussion because of its impact on estimation of life expectancy. We must emphasise here that WHO has not undertaken any new adjustment to the officially reported All Cause Mortality (ACM) for Indian population. What has been revised is the officially declared mortality due to COVID-19. Estimation of Life Expectancy during the pandemic years critically depends on a correct estimate of death due to  COVID-19 for the simple reason that such deaths were much higher in the adult age-groups. Any spike in the mortality of higher age groups would then bring down the life expectancy at birth. Bhattacharya et al has pointed out how social stigma associated with segregation of COVID-19 patients led to a situation of “creating a fear among the public and is acting as a deterrent to the effective management of the disease, particularly in the urban setup (Bhattacharya Prama er al 2020).” So, there is a distinct possibility of under reporting of COVID related deaths.

Since COVID-19 is a new phenomenon, death due to COVID-19 would only add to number of deaths that would have happened in the absence of COVID-19. WHO has defined this excess death as “the mortality above what would be expected based on the non-crisis mortality rate in the population of interest”.

WHO has identified two countries (India and Indonesia) for which ACM data is available only at sub-regional level and not at national level. For these two countries, WHO constructed “a multinomial model, based on the assumption that the fractions of deaths in sub-regions remain approximately constant over time.” (Knutson Victoria et al 2023). Excess death estimated by WHO’s method has been compared with the results of few other methods and differences are within reasonable bounds.

Approach	Estimate (10^6)	95% Confidence interval	Period
Naïve	5.04	(4.48,5.59)	Jan20-Dec21
WHO	4.74	(3.31,6.48)	Jan20-Dec21
The Economist	4.86	(1.70, 8.47)	Jan20-Dec21
IHME	4.07	(3.71, 4.36)	Jan20-Dec21
Naïve	4.29	(4.00,4.59)	June 20-June21
WHO	4.33	(2.85,6.13)	June 20-June21
Jha et. al.	3.23	(3.06, 3.39)	June 20-June21
Naïve	3.96	(3.62, 4.29)	April20-June21
WHO	3.99	(2.40, 5.95)	April20-June21
Anand et.al. 2021 Method 1	3.4		April20-June21
Anand et.al. 2021 Method 2	4		April20-June21
Anand et.al. 2021 Method 3	4.9		April20-June21

A few researchers have dealt with this issue of estimating death rates directly attributable to COVID-19 using micro level sample data. I would like to refer two studies for their innovative use of covid related death rates of well demarcated community of people to work out a national level estimate of death due to COVID-19. The first study was by Christophe Z. Guilmoto (2022), who identified four different samples of Indian populations, having complete data on their demographic structure and death occurrence. Selection of the four samples were largely subjective and not driven by any sampling mechanism. However, the author had applied three criteria for sample selection. These were – reliability of death estimation, regional   representativeness, and demographic characteristics. The selected samples were:

a: Kerala sample: Dataset of deaths by age and sex in Kerala state

b. MLA sample: Deaths of elected representatives

c. IR(Indian Railways) sample: Deaths of Indian Railways personnel

d. Karnataka sample: Deaths of schoolteachers in Karnataka state

The Kerala sampleis a dataset with 26,628 Covid-19 deaths classified by age and sex on

18 October 2021. The MLA sample is relatively small sample of 5,837 MPs and MLAs. The Karnataka sample is a sample of 268 deaths attributed to Covid-19 among 196,163 schoolteachers of Karnataka.

The author followed the following estimation strategy of the country level mortality due to COVID-19.

1. Use Kerala data to determine the age and sex schedule of Covid-19 mortality in

India, but not to estimate its intensity.

• Apply Kerala Covid-19 mortality rates to project the number of deaths in the three other samples (MLA, IR and Karnataka). Intensity of mortality is adjusted to fit the number of observed deaths in these three samples.
• This adjusted mortality patterns is applied to India’s age and sex structure to yield national level estimates of Covid-19 mortality in May 2021. The numbers of deaths are finally projected to 15 October 2021 using trends reflected in the official Covid-19 casualty statistics and contrasted with international figures using standardized rates

The following table summarizes outcome of this exercise.

Table 3.3 : Estimates of cumulative Covid-19 deaths in India

	Sources			Units	Reference day           (in 2021)
	Official estimate*	IR Sample	MLA sample
Covid-19 deaths	243.5	1,701.80	—	000s	07-May
	325.8	2,276.60**	2,597.10	000s	25-May
	458.9	3,206.90**	3,658.30***	000s	1st November
Correction factor	1	7	8.6	—-	1st November
Death rates	0.3	2.3	2.6	per 1000	1st November

Acosta et. al. (2022) have estimated excess mortality in the State of Gujarat, India, during the COVID-19 pandemic (March 2020-April 2021) by using data from civil death registers from a convenience sample of 90 municipalities across the state of Gujarat. This study found that “the largest increases in mortality occurred in the second wave of the pandemic, across all municipalities”. It was further found that “over 80% of the municipalities experienced increases in mortality over 100% in all demographic groups except the two youngest age cohorts. Lastly, in over 50% of the municipalities, males and the 40 to 65 years groups experienced an increase of over 500%.”

To sum up, there is enough evidence to accept the Life Expectancy estimated by WHO for the two pandemic years. The suggestion by the authors of Reversing the Gaze, “that the Registrar General of India should timely publish life expectancy estimates every year” is highly recommended but not for the alleged unfounded observation that “one-sided adjustments and circular references are routinely done to India-related data by international agencies.”

Section 2: Analysis of Methodological criticisms of WP 2

This paper’s stated objective is to provide a “quantitative analysis of the data quality of the National Sample Survey (NSS) in terms of three estimates, (i) the proportion of the rural population, (ii) the proportion of the Scheduled Caste (SC) population, and (iii) the proportion of the working-age (age between 15 and 59 years) population”.

Although the above noted estimates are published in the reports of these surveys, these are neither the focus of any NSS survey nor are these used in any survey report from this perspective.  On the contrary, population census data is used for allocation of total sample to various stages of selection of survey households at village and urban block level[1]. To be consistent with survey design, it is necessary to estimate population from the selected households and work out various ratios like labour force participation rate, distribution of persons of each sector of each State/UT over 12 classes of MPCE (monthly per capita consumption expenditure).

---

[1] NSSO follows a stratified multi-stage design for its surveys. The first stage units (FSU) are the 2001 census villages (Panchayat wards in case of Kerala) in the rural sector and Urban Frame Survey (UFS) blocks in the urban sector. For the rural sector, the list of 2001 census villages constitutes the sampling frame. For the urban sector, the list of UFS blocks (2007-12) is considered as the sampling frame

Notwithstanding the above, drawing direct comparisons between the estimated rural and urban population data obtained from this survey and the corresponding figures from the 2011 census data requires careful consideration due to several factors. Firstly, the census figures were collected as on a particular day- 31st March 2011, whereas this survey was conducted in 4 sub-round of 3 months each, spanning July 2011 to June 2012. The figures from the survey are not expected to match with that of census data. Secondly, there are homeless people, convicted prisoners in jail, floating populations who are not covered by NSS surveys. Such people would be more likely to be in urban areas and this would negatively impact share of urban population. Nevertheless, there is no straight forward answer to the impact that these exclusions would have on the target measures like rate of unemployment or incidence of poverty at sub-national levels.

Keeping the above caveat in perspective, I argue in the following paragraphs that the application of the new concept of Total Sample Error (TSE) is a completely misguided application of statistical techniques which have been specially developed for sampling issues related to Big data and/or large non probabilistically obtained administrative data.

Professor Meng has clarified that his paper is focussed on population inferences from Big Data (Meng 2018). Dr. Ravi and her colleagues have used the formula given in the Meng’s paper to compute sampling bias- the difference between the sample average of a variable of interest and its population average . This has been worked out, in Meng’s paper, as a product of three terms: (1) a data quality measure, (2) a data quantity measure, and (3) a problem difficulty measure. In case of NSS survey, the data quality is to be ascertained at the final stage sampling process- from a village or an urban block because final estimates are built up from that level. There is no evidence so far that NSS surveys suffer from a data quality issue because respondents are non-cooperative or deliberately providing wrong numbers. Furthermore, implementation of interpenetrating subsample and use of two sets of investigators (central sampling and state sampling) would clearly bring out any abnormal variation in estimation of target variables. In fact, Dr. Meng agrees that “under a genuine probabilistic sampling, the chance that a particular value of G (i.e., target variable, my addition) is recorded/reported or not should not depend on the value itself. Consequently, ρR,G should be zero on average.” So would be the sampling bias.

In another paper on this subject, the same inference has been drawn for a probabilistic sample survey, – “random sampling or any sample design that provides constant inclusions probabilities, the term for data quality ER(𝜌RX) is 0 and thus the bias is also 0 (Biemer and Amaya 2018)

Foot note: NSSO follows a stratified multi-stage design for its surveys. The first stage units (FSU) are the 2001 census villages (Panchayat wards in case of Kerala) in the rural sector and Urban Frame Survey (UFS) blocks in the urban sector. For the rural sector, the list of 2001 census villages constitutes the sampling frame. For the urban sector, the list of UFS blocks (2007-12) is considered as the sampling frame

Unfortunately, Dr. Shamika and her colleagues have mechanically used Dr. Meng’s approach developed for Big data/Administrative data to assess quality of NSS survey data, without critically reviewing the NSS sampling design. The way to have an independent assessment of any sampling estimate was nicely brought out with an example by none other than founding fathers of NSS – Prof P. C. Mahalonobis and D. B. Lahiri. In their paper (1961) “Analysis of Errors in Censuses and Surveys with Special Reference to Experience in India” they compared the “results obtained on the basis of complete enumeration and sample survey against a third, but extremely reliable, figure.” This could be carried out for two consecutive years – 1944-45 and 1945-46- for Jute Crop of Bengal. Jute being a cash crop of international importance, accurate export trade figures were available after a gap of 15 months of harvest. So, there was a need for a quick estimation of production. The official approach to meet this demand was to carry out a plot-to-plot enumeration. Indian Statistical Institute undertook two sample surveys- one for estimation of acreages and subsequently another for post-harvest yields. Surveys were conducted using two interpenetrating sub-samples (IPNS) which were covered by different parties of investigators. These two independently computed values were compared with the trade figures. The result was follows.

” in both the years the official forecasts based on complete count were both very much out whereas the sample survey estimates were quite close to the trade figures. The two sub-sample (IPNS) estimates in both years agreed with the trade estimates within roughly 3 per cent while the estimates based on the so-called complete count differed from the trade figures by 27.2 per cent in 1944-45 and 16.6 per cent in 1945-46”

Finally, it would be amiss if Dr. Ravi’s stringent criticism of NSS sampling design in her article “The sample is wrong” published in the Indian Express 7th July edition. Three areas of her concerns are: availability of data, transparent and robust statistical analysis, and data quality.

The first area of concern- more frequent and timely availability of survey data to policy makers for evaluating any policy intervention at national level- is indeed a crucial one, and it requires a coordinated effort between statistical authorities and government decision-makers to address it effectively. As a member of the EAC-PM the author can play a significant role in advocating for the importance of timely and frequent survey data.

As I have pointed out in my discussion on WP-2 above, the data quality measure used therein is meant for non-probabilistic data or for an integrated dataset of administrative data and a probabilistic sampling data. Thus, the author has not given any statistically valid argument to declare that “Sample is wrong”.

Nevertheless, the author’s use of Meng’s approach to measuring of Total Sampling Error has inadvertently brought into focus the need for using “found data” in conjunction with the usual probabilistically sampled data for estimation of many social and economic indicators of a country as large as India. Administrative data are a type of found data which gets routinely generated as a part of the official activities like getting a subsidy, registering of birth and death, registering of vehicles, getting unemployment benefit, income tax data etc.   Many official statistical authorities in advanced countries are already exploring big data or found data integration with survey data to achieve better quality of statistical estimates of important economic and social indicators. For example, Statistics New Zealand has established an Integrated Data Infrastructure (IDI) , sourcing data from government agencies, Stats NZ surveys, and non-government organisations (NGOs). Statistics Netherlands has established a Center for Big Data Statistics in 2016 “with the primary purpose of leveraging new and existing (big) data sources and techniques in order to arrive at better information about social themes such as the labour market, mobility, health, the energy transition and smart farming.” (Unique collaboration for big data research (cbs.nl) ). Statistics Canada uses administrative data to complement or to replace survey data.

Dr. Ravi’s criticism about the quality of Indian Official Statistics, however methodologically flawed it might be, needs deep appreciation because it has highlighted the absence of any innovation in statistical methodology used in production of the critical social and economic indicators. in the last two decades. Thus, to paraphrase Agatha Christie, the Indian Statistical Mirror as shown to us by the official statistical system might not have “crack’d from side to side”, but it is gradually becoming covered with dust of obsolescence.