Family Tree

Family Tree

About Me

My photo
Kathmandu, Bagmati Zone, Nepal
I am Basan Shrestha from Kathmandu, Nepal. I use the term 'BASAN' as 'Balancing Actions for Sustainable Agriculture and Natural Resources'. I am a Design, Monitoring & Evaluation professional. I hold 1) MSc in Regional and Rural Development Planning, Asian Institute of Technology, Thailand, 2002; 2) MSc in Statistics, Tribhuvan University (TU), Kathmandu, Nepal, 1995; and 3) MA in Sociology, TU, 1997. I have more than 10 years of professional experience in socio-economic research, monitoring and documentation on agricultural and natural resource management. I had worked in Lumle Agricultural Research Centre, western Nepal from Nov. 1997 to Dec. 2000; CARE Nepal, mid-western Nepal from Mar. 2003 to June 2006 and WTLCP in far-western Nepal from June 2006 to Jan. 2011, Training Institute for Technical Instruction (TITI) from July to Sep 2011, UN Women Nepal from Sep to Dec 2011 and Mercy Corps Nepal from 24 Jan 2012 to 14 August 2016 and CAMRIS International in Nepal commencing 1 February 2017. I have published articles to my credit.

Saturday, May 9, 2020

Exploring Quantitative Data


Exploring data is a kind of data health check to see whether there are outliers that cause abnormal distribution. Parametric inferential statistics are designed to fit the normal distribution. Several summary functions and graphical displays are used to explore data. I exemplify these measures from the daily database on COVID-19 incidence worldwide as of April 29, 2020, which I downloaded from the European Centre for Disease Prevention and Control website. The dataset has 109 daily records. This article needs to read along with my previous article entitled ‘Describing Quantitative Data’.

Five Number Summary

Minimum value, first quartile (Q1) or 25th percentile, median or 50th percentile, third quartile (Q3) or 75th percentile and maximum value are five summary numbers used to explore data. In a normal distribution the median is at the center of the distribution. Besides, the difference between the minimum value and median is equal to the difference between the median and maximum value. Likewise, the difference between the minimum value and Q1 is equal to the difference between Q3 and maximum value as shown by black and red arrows in figure 1.

Figure 1: Five Number Summary in Normal Distribution






In positively skewed distribution distance between minimum value and median is smaller than the distance between median and maximum value as shown by arrows in figure 2. Likewise, the distance between the minimum value and Q1 is smaller than the distance between Q3 and maximum value.

Figure 2: Five Number Summary in Positively Skewed Distribution






In negatively skewed distribution the distance between minimum value and median is greater than the distance between median and maximum value as shown by black and red arrows in figure 3. Likewise, the distance between the minimum value and Q1 is greater than the distance between Q3 and maximum value.

Figure 3: Five Number Summary in Negatively Skewed Distribution





In the given dataset, the minimum value is one and the maximum value is 101,728. Q1, median, and Q3 are respectively 1761.5, 3907, and 66403.5. The difference between minimum value and median is equal to 3906 which is far smaller than the difference between the median and maximum value equal to 97,821.  This is consistent with figure 2 indicating that data is positively skewed.

Normality Checks

Box plot displays the five number summaries as shown in figure 4. The thick black line inside the box symbolizes the median. The lower and upper hinges or boundaries of the box symbolize Q1 and Q3, respectively. Whiskers below and above the hinges symbolize the minimum and maximum values. Box plot shows the outliers and extreme values. Outliers are the values below or above the value calculated as one and half times the difference between Q1 and Q3. The given dataset does not have statistically outlying values as there are no indications on the box plot. However, there are five extreme values above Q3 from 86,046 to the maximum number of persons infected in a day. Likewise, five extreme values below Q1 from 17 to the minimum number of infected cases. Those values have led to skewed distribution.























The stem and leaf plot shows the shape of distribution showing positive skew statistic in which the values cluster below 10,000 cases per day having 62 data points. Values from 50,000 to below 90,000 cases are also clustered as shown in figure 5. This shows that the distribution is positively skewed.

Figure 5: Stem and leaf plot













A normal Q-Q plot displays the straight line that represents expected values for normal distribution. The observed incidence values deviate distinctly from that line indicating positively skewed distribution as shown in figure 6.

Figure 6: Normal Q-Q Plot























In a nutshell, exploring data summarizes distribution, identifies outliers, and checks normality. These procedures confirm that the distribution of the given dataset is positively skewed.

Tuesday, May 5, 2020

Descriptive Measures of Quantitative Data

Descriptive statistics measure central tendency, dispersion, and distribution. I exemplify these measures from the daily database on COVID-19 incidence worldwide as of April 29, 2020, which I downloaded from the European Centre for Disease Prevention and Control website. The dataset has 109 daily records.

Measure of Central Tendency

Mean incidence per day is 28,016.6 persons. Median or second quartile is located at 3,907 persons and mode is 17 persons infected in a day. Mean, median, and mode are equal for normal distribution. Both median and mode are less than mean indicating that the dataset is distorted from normality.

Measure of Dispersion

Range of 101,727 is very wide between a maximum of 101,728 persons to one person infected on a single day worldwide. The standard deviation is calculated at 34,079.4 persons. For the normal distribution, 95 percent of the data values fall within two standard deviations from the mean. In this dataset, 95 percent of data values fall within plus 96,175.4 to minus 40,142.2. 























Measures of Distribution

Skewness value is positive 0.73.  A normal distribution has zero skewness. The standard error of skewness is 0.23. The positive value indicates that the distribution is positively skewed with a right tail. Since a skewness value is more than two times its standard error the distribution is asymmetric.

Kurtosis value is negative 1.24. A normal distribution has a kurtosis value of zero. The standard error of kurtosis is 0.45. The negative value indicates that the distribution is flatter than normal with the data values less clustered around the center of the distribution, which is also referred to as platykurtic distribution. Since a kurtosis value is more than two times its standard error the distribution is abnormal.

In a nutshell, descriptive statistics measure the central tendency, variability, and distribution of data. The given dataset is positively skewed and flatly distributed than normal. 

Saturday, May 2, 2020

Frequency, Tabulation and Graphical Presentation

Frequency is a summary measure that describes both categorical and quantitative data. It counts the number of times a category or a data value is repeated. The summary data can be presented in tables and graphs. I exemplify from the database on COVID-19 incidence and human death worldwide as of April 28, 2020, which I downloaded from the European Centre for Disease Prevention and Control website.

Categorical Data
I was curious to know how many countries in different continents or regions had had deaths.  I found that 169 out of 206 countries had deaths and 37 did not deaths as shown in Table 1. Most European countries had deaths and most African countries did not have deaths.

Table 1: Countries Reporting Death Due to COVID-19



















The above data can be visualized in a bar chart for better presentation as shown in Figure 1.











Quantitative Data

I was curious to know how many countries had how many numbers of death due to COVID-19 falling in a certain class interval. I found that most countries (161 out of 169) reporting death had deaths falling within a class interval of one to less than 5000.  It was followed by three countries that had deaths in the interval between 5,000 to less than 10,000. One country had deaths within the range of 55,000 to less than 60,000 (Table 2).

Table 2: Country and Death Due to COVID-19


















The above data can be presented in the histogram as shown in Figure 2. Data shows that the number of countries is positively skewed.








I was curious to know how many persons died at what time interval. It was appraised that in April 2020 alone 82 percent of total 209,776 people died. It seems that the number of deaths is rapidly increasing by the month. It could be because people who suffered in initial months started to die.

Table 3: Death by Month in 2020
Above data can also be visualized in the Time series line chart in Figure 3.









In a nutshell, frequency is a process of summarizing data, which can be presented in tabular and graphical forms.

Wednesday, April 29, 2020

How Predictable Death by COVID-19 Incidence?


I downloaded from the European Centre for Disease Prevention and Control website the daily database constituting the number of new cases reported and death worldwide due to COVID-19. The database had 13,623 daily records from December 31, 2019 to April 28, 2020. I summed records by country and filtered the number of incidences avoiding outliers using a statistical formula.  The filtered record shows a list of 136 countries that had an incidence in 314,099 persons, of which 7,866 persons died. It shows that 2.5 percent of infected people had already died and there is a possibility of other deaths too, which the time will tell. Israel had a maximum of 15,598 incidences, and the British Virgin Islands had a minimum of 6 incidences. Algeria had a maximum of 437 deaths and 13 countries each had one death.

Then, I calculated the Pearson correlation coefficient between the number of incidence and deaths for the filtered records.  The purpose of removing outliers was because the Pearson correlation coefficient works when the datasets are close to normal distribution. The analysis shows a correlation coefficient of 0.677 which was significantly different from zero (p=0.01). This suggests that there is a moderate chance of death if one is infected with COVID-19. A regression analysis was also conducted to predict death based on incidence, which gives an R square value of 0.459 indicating that nearly half the variation in the number of deaths is explained by the model. A significant coefficient indicates that the number of incidences significantly contributes to the model. For every 1000 incidences 18 deaths are likely.

Thursday, April 9, 2020

Graphics Visualizing Target Population

Graphics is an important form to visualize one data from others. This technique is useful to visualize the target population for any purpose. For example, I referred to Nepal’s population census 2011 data to get population values to visualize the number of male household heads from urban areas of Nepal belonging to the age group of 50 years and above. I used four criteria to filter and get target population data – male, age group, administrative setting (urban), and position or role in a household (household head).
An outermost blue circle shows the total population of Nepal, which is 26,494,504. It is followed by a red circle showing the male population which is 49 percent of the total population. An inner third yellow circle shows the male population aged 50 years and above, which is 15 percent of the total male population. An inner dark ash-colored circle shows the urban male population aged 50 years and above, which is 15 percent of the total males aged 50 years and above. Lastly, an innermost green circle shows the number of urban male household heads aged 50 years and above, which is 81 percent of total urban males aged 50 years and above and 1.9 percent of the total male population in Nepal.
These data are visualized in both stacked Venn and horizontal hierarchical diagrams in Figures 1 and 2 respectively.

Figure 1: Target population of urban male household heads aged 50 years and above shown in a stacked Venn diagram


Figure 2: Target population of urban male household heads aged 50 years and above shown in a horizontally hierarchical diagram

In a nutshell, visualizing filtered data in different diagrammatic forms is a cool way.

Monday, December 2, 2019

Level of Confidence Increases with Sample Size: An Example of One Sample Proportion, Statistical Note 47

Level of confidence increases with an increase in sample size for one sample proportion. This note tries to exemplify this fact by using the process and data discussed in my statistical note 43 and 44 respectively.

For example, an expert is interested in knowing the proportion of non-smokers from the randomly sampled respondents (following randomization, the first rule of sample proportion). An expert assumes that half of adult population are smokers. An expert administers a question to the randomly adults – Are you a smoker? The respondents answer to one of two categories of response – Yes or No.

An expert tries with a sample size of 100 adults and finds that 55 are non-smokers and remaining 45 are smokers (following normality that non-/smokers need to be at least 10, second rule of sample proportion). An expert then retains the same sample proportion of smokers and hypothetically increases the sample size by 100 to 500. An expert assumes to follow independence in sampling without replacement that the population size is more than 10 times of sample size, third rule of sample proportion.

An expert estimates how confident he is in deciding that non-smokers statistically outnumber the smokers with the increase in sample size. An expert uses the following formula to calculate the minimum number of non-smokers from the given sample size to outnumber the smokers:

 z=(p^-p)/√(pq/n)
  where ,
z=Test statistic, standard normal variate, with a value of 1.96 at 95% level of confidence
p^=Sample proportion of non-smokers
p=Population proportion of non-smokers, equal to 0.50
q= Population proportion of smokers, equal to 0.50
n=Sample size

Table 1: Sample size with Same Sample Proportion of Non-Smokers and Level of Confidence to Conclude Non-Smokers Outnumber Smokers











An expert is 84 percent confident in deciding that the sample non-smoker proportion of 0.55 among 100 respondents statistically outnumbers the sample smoker proportion of 0.45. Usual threshold is that decisions are made at 95 percent level of confidence. Gradually, an expert increases sample size by 100 and finds that for a sample of 300 respondents or more an expert is more than 95 percent confident in deciding that the sample proportion of non-smokers equal to 0.55 is statistically higher than the sample proportion of smokers equal to 0.45. It proves that the level of confidence increases as sample size increases for one sample proportion.

Wednesday, November 27, 2019

Conditional and Joint Probabilities from an Exemplary Survey Dataset, Statistical Note 46

Understanding the concepts of joint and conditional probabilities and developing the skill to apply the concepts to calculate from the given dataset is important in the real time.

An exemplary survey dataset constitutes one hundred records of randomly sampled respondents categorized by smoking habit (smokers or non-smokers) and food habit (vegetarians or non-vegetarians). A part of the dataset in value label view of SPSS is shown in Table 1. Calculate the probability that a randomly sample respondent is a non-smokers is a non-vegetarian.

Table 1: Part of a dataset with smoking and food habits


























Concept

Calculating the probability of a "AND" compound event that a randomly selected respondent is a smoker who is a vegetarian also includes calculating the probabilities of other events. Several concepts are introduced while answering this question.

This example has two discrete random variables or categorical variables each with two mutually exclusive categories of response. One categorical variable is the smoking habit of a randomly sampled respondent which has two categories of response: smoker (S) or non-smoker (NS). Another categorical variable is the food habit which also has two mutually exclusive categories: vegetarian (V) and non-vegetarian (NV).

Simple or marginal probability: Let ‘S’ be a random event that a randomly sampled respondent is a smoker. The probability of randomly sampled smoker, represented by P(S) is the total number of smokers divided by total number of respondents. It is also referred to as the relative frequency. Similarly, P(NS), P(V) and P(NV) are calculated.

Conditional probability: Let V/S be a simple event that a participant is a vegetarian among the smokers. The conditional probability of vegetarians among smokers symbolized by P(V/S) is total number of vegetarians among smokers divided by total number of smokers. Here the occurrence of the event of vegetarian smoker is dependent on the event of occurrence of smokers.

Joint Probability: Let ‘S intersection V’,  ‘S∩V’ or ‘S and V’ is a "AND" compound event that a respondent is a smoker and a vegetarian. Here, the multiplication rule of two dependent events is applied. The joint probability of two dependent events is the product of a marginal probability and the conditional probability. In this case, the joint probability of a smoker who is a vegetarian indicated by P(S intersection V), P(S∩V) or P(S and V) in which both events of smoker and vegetarian among all smokers occur is the product of P(S) and P(V/S). Likewise, P(NV/S), P(V/NS), P(NV/NS), P(NS∩V), P(S∩NV) and P(NS∩NV) are calculated.

Calculation

A contingency or cross table from the given survey dataset can be generated either in SPSS or Excel package upon the availability of the software.  In SPSS, using the function ‘Crosstabs’ in Descriptive Statistics’ group of ‘Analyze’ tab, one can get the cross table as in Table 2. In Excel, ‘Pivot Table’ function in the ‘Tables’ group in ‘Insert’ tab can be used to generate cross table like this.

Table 2: Cross table of smoking habit and food habit
















Table 2 constituting four cells and totals to presents the frequencies, row percent (% within SMOKE), column percent (% within VEG) and percent of total respondents. Now, the concepts discussed above are applied to calculate probabilities.

Simple or Marginal Probability: Table summarizes that 25 out of 100 respondents are smokers so that P(S) is equal to 0.25, which is 25 percent in percentage term as shown by ‘% of Total’. P(NS) is 0.75 or 75 percent in percentage term. Likewise, P(V) is 0.25 or 25 percent in percentage term and P(NV) is 0.75 or 75 percent in percentage term.

Conditional Probability: P(V/S) is calculated looking at the first cell of the table. Eight out of 25 smokers are vegetarians so that P(V/S) is equal to eight divided by 25 equal to 0.32, which is equal to the row percent (% within SMOKE) of 32% in percentage terms. Similarly, other conditional probabilities are calculated as P(NV/S)=0.68, P(V/NS)=0.227, P(NV/NS)=0.773.

Joint probability: P(S∩V) is the product of P(S) and P(V/S), equal to the product of (25 by 100 ) and (eight by 25), equal to 0.08 or 8 percent in percentage term. This is equal to ‘% of Total’ in the first cell of Table 2.

Upon filling manually the probability values in yellow highlights of Table 2, the table looks as Table 3. The joint probability for each cell is equal to percent of total value in percentage term.

Table 3: Cross table of smoking habit and food habit with probabilities manual added





















It is hoped that such a simple example will help create curiosity among the readers as to applying the concept in the real time data.