In this chapter, you will become familiar with the R and Python languages. Here, R and Python will be used in parallel, but the rest of the book may not always follow that pattern; there may be a situation in which R would seem simpler to use than Python or a situation in which Python would seem like the wiser alternative. However, I need to be clear here: the choice of R or Python for a particular task does not mean that the task cannot be done in the option not chosen. It simply means that it would be harder, less elegant, or too confusing to use the other language, or it might require the use of techniques and concepts not covered in this book.

First Contact with R and Python

You will need first to locate the RStudio and Anaconda environments in your computer. The icons are shown in Figure 3.1.

Becoming Familiar with R

Let's start running RStudio first, as shown in Figure 3.2. In this case, I decided to change the position of the RStudio of the windows, so the source is to the left and the console is to the right (the other windows are not shown). The console is showing only the first message you see when you start RStudio. In your case, the source window should be empty, but mine is not. Please write the code as it is shown in Figure 3.2. Keep reading after you finish typing the code.

Let's use the numbers that RStudio offers in the source window to review the code you have typed:

1. • Lines 1 and 15 are very special in RStudio. They create a code section, several lines of code that run together. It is a good practice to organize your code in sections: you need to structure your work to improve it progressively and facilitate debugging. A section starts with the symbol #; then you write the name of the section, and finally you add at least four dashes, so that the section is recognized as such. You can quickly visit the sections you create using the drop-down list you find in the bottom corner to the left of the source window and selecting the label of the section you want. In Figure 3.2 you can see that the label “Calculator” appears in that area, close to a pound symbol.

This book is based on class notes used to teach undergraduate and graduate students in political science and public policy how to prepare their data to conduct further analysis and provide recommendations to inform decision making. At both levels of education, I found students with different backgrounds in quantitative tools. That required me to develop a teaching approach to address the needs of students new to data analysis, so you should expect detailed explanations in this book.

My courses address different skills: from collecting, organizing, modeling, and analyzing the data, to visualizing and publishing it. This book focuses on data organization and collection and all their related procedures. This stage is arguably the most time-consuming and has been my main concern over the years. In general, books for quantitative analysis devote most of their pages to teaching quantitative concepts while sharing data that has already been well organized into a table readable by most software (Stata, SPSS, SAS, etc), or into a simple comma-separated values (CSV) file to be used by any software. However, every scholar (student, researcher, or professor) or professional will need to prepare a particular data set for his or her current needs. It is then that they may experience a hard time. The literature lacks material on preparing data sets for students in the social sciences, so it has been my goal to provide those competences in basic data organization.

When I embarked on teaching this stage to students, I discovered there is no simple, right, or unique way of doing it. In this book, I simply share my way of teaching it, not how it should necessarily be done. This is a potential challenge or weakness of this book because I do not present a formal paradigm for data organization. Nevertheless, I share strategies that have proven to be helpful for students who lack a quantitative or computational background.

My target audience might be familiar with particular software, but may never have used it to collect or clean data or to automate those processes. There is no function in an SPSS/Stata/Excel window menu that says either collecting or cleaning the data. Collecting, cleaning, and formatting data require the flexibility that programming languages provide, which motivated me to teach some coding skills to my students.

Data cleaning is an iterative process. However, this chapter is more concerned with explaining the cleaning steps than the order in which you should do them. It reviews each data frame created in Chapter 4, performs a basic data exploration, and then cleans the “dirty” values, – those that have some anomalies caused during the process of importing the data into R or Python or because the original values were not ready to be imported into a different software. I focus on missing values, encoding, and, especially, symbols that were not imported correctly.

Dealing with Missing Values

Missing values are likely to be present in every data set. Data related to engineering activities can get lost, but, in general, a missing value indicates that a sensor did not record a particular data value. In social data, however, you will not only find an “NA” representing missing values but they may be coded with other values representing varieties of “missingness.” The coding of missing values is well covered by licensed software such as STATA or SPSS, but not explicitly covered in standard R or Python. As we learned in the last chapter, the memisc and foreign packages can import SPSS data very well into R (although there is less support from Python) and that Python and R can import without problems STATA files and the other formats we explored. Let's see how Python and R deal with missing data.

Identifying and Setting Missing Values: Codebooks

Finding a data set without missing values is rare. The problem is knowing whether the missing values are clearly identifiable or not. In general, if your data set comes with a codebook, then that means that you will know all the metadata needed. If you scrape the data, you may not know if you are missing values or not.

The SPSS data from ANES is a good example of the varieties of missing data in social science research. Table 5.1 from ANES time series for 2012 shows clearly which values should not enter into the calculations, but are nevertheless informative. That table was created using the codebook provided by ANES:

Table 5.1 has many values that should be considered missing values, and we have the codebook to identify them.

It is time to start working. Here, I share with you the ways in which Python and R, and their associated tools, should be installed on your computer. For most of this book, and for this chapter in particular, you need to be connected to the Internet. If you are a Windows, Linux, or Mac OS X user, each installation requires the usual steps for your system, which in general requires that you accept every option; however, I give you some suggestions when necessary.

Installing R

To download R, you need to visit https://cran.r-project.org/. That webpage clearly indicates the options for Windows, Mac, or Linux users. Windows and Mac users need to visit the appropriate link and get the file you need, checking if the version offered is the right one for your operating system. As Linux users may expect, after they go to the Linux link, R will ask you to choose among the Debian, redhat, suse, or ubuntu options.

After you download the file from R, you need to install it. This is simply done by clicking the file you downloaded to your computer. Just run it and keep all the default settings. The Linux versions have particular instructions to run in the terminal once the right file is chosen. You do not need to run R now (if you opened it, just close it). We will run R always from RStudio.

Installing RStudio. To get RStudio, just go visit the downloads website.

Once in that webpage, you will find links to the different desktop versions available (as opposed to the server versions, which you do not need). The webpage with the installers looks like Figure 2.1. Download the version you need according to your machine, install it, and accept all the default options. Now, when you start RStudio, you will see a graphical user interface (GUI), composed of a set of windows as shown in Figure 2.2.

The RStudio GUI has four windows. You may have a different set of windows in RStudio, so Figure 2.2 should be only a reference.

This section describes how to use R and Python to help you with the second step of the roadmap in Section 1.1 on page 4. You are taking this step because you need to get some data. However, this step assumes that you already have defined a problem you want to understand better, have laid out some hypotheses, and have identified relevant data sources, as discussed in Section 1.1; those tasks drew more on your critical thinking skills than your computational skills.

So, now you are in a situation where you know what data to collect and from where. The source can be a webpage or a file to be downloaded. Most of the time you will get files, so make sure you understand exactly what each variable means and how the values are represented in the file. This requires that you read the documentation of the data file, paying particular attention to the methodology used to collect the data.

Knowing Where Your Files Are

None of the examples in the previous chapters imported a data file, but for the rest of the book, we will only work with real data files. I recommend that from now on, you keep all your files in Dropbox, as I suggested in Section 2.3 on page 22. The data for this book is in a folder named BookData, and you can use this shortened link to download it: https://goo.gl/Czi3Vh. The screen will look similar to Figure 4.1.

If you have installed Dropbox in your machine, you only need to select the option Save to my Dropbox, which will appear after you press the Download button in the right upper corner, as also shown in Figure 4.1. I strongly recommend placing the “BookData” folder in the Dropbox root folder (I assume you have done so in my examples). If you do not want Dropbox, just go for the option Download as .zip and then unzip the folder in the root of your computer.

Next, tell RStudio that you want to be in the “chapter4” folder and that you also want all your code to be saved there.

In general, when you collect, clean, and format data, you are working with independent files, but most of the time your analytic work needs to combine files from different sources to answer the questions you define in your research design. It is rarely the case that your research question depends on one particular data set or on several files from one source. Integration is the process of combining your data to produce the file you need. The material on storage focuses on how to code for data retrieval, updating, and sharing; topics that go beyond simply saving your files for later use.

Integrating Data

Preparing the Integration: Cleaning the Keys

To be able to integrate different data sets, you need one column that serves as a key to merge the data. Sometimes, you may even need two columns. That key column is the common column in every data set to be integrated. The headers do not have to be the same (no need for that), but the contents do. For example, if you have two data files of cities from different countries, the key could be the city name. If one file has more cities than the other, it is not a problem, as long as the cities that appear on both files are written using the same spelling. For this example, I will combine some known data sets from different sources to show the issues you face when integrating data.

1. • Human Development Index (HDI)

2. • Freedom of the World Index (WORLD)

3. • Economic Freedom Index (ECO)

4. • Freedom of the Press (PRESS)

5. • A table with the country names and their ISO and FIPS (CODES) from America's Open Geocode (http://www.opengeocode.org/) database

The unit of analysis is the country; the column with the country names can be used as the key, but I prefer some kind of country code, so the ISO or FIPS codes are the best option to ease the merging. Notice that I will use the official link to the data this time (the ones provided in the previous footnotes) to avoid scrapping (and mistakes in Wikipedia information). Let's start by getting the ISO and FIPS codes.

INTRODUCTION

In this chapter, we discuss four basic sampling designs, namely, simple random sampling (SRS), stratified sampling, systematic sampling and cluster sampling. In the process of sample selection, it is possible to select elements directly, if a suitable sampling frame for their selection is available. Often, sampling frame is available only for a group of individual elements, instead of each element. These groups of individual elements, generally some areal units, are commonly referred to as clusters. If the sampling frame for each cluster in a population is available, the sample can be selected either in one stage or at multiple stages. In one stage, in a selected cluster, all the elements comprising it will be automatically included in a sample. Selection in two or more stages, discussed in Chapters 3 and 4, involves selection of elements from selected clusters.

Of the four designs, mentioned above, SRS and systematic sampling are the two basic methods for selection of sampling units (either units or clusters). Both provide equal chance to each sampling unit for inclusion in a sample. Another variant, known as PPS sampling wherein sampling units are selected with varying probability, is also elaborated here. Further, we discuss stratified sampling which is a technique of grouping sampling units in a population before selection where grouping is done to put restrictions on selection of a combination of sampling units to benefit the procedure of selection. The cluster sampling in which all the elements in a selected cluster are included in a sample is generally avoided. However, it facilitates understanding multistage designs. In each of the next four sections, we outline the essential steps in a design. They are as follows:

1. • Methods of selection of sampling units,

2. • Estimation of a parameter, and

3. • Providing estimate of the error such as sampling variance of an estimate.

SIMPLE RANDOM SAMPLING

2.2.1 Description

In this design, each and every sampling unit in a population receives equal chance of being included in a sample. The method is discussed assuming individual units as the sampling units, but remains true even if group of units, instead of individual units, is considered. Let there be N units in a population from which n are to be selected for inclusion in a sample.

The twenty-first century is the Century of Information. Scientifically designed gathering, collation, analyses, interpretation, sharing and dissemination of information have become the primary occupation of millions, as the activity encompasses the widest possible spectrum of subjects ranging from education to development including social sciences, developmental economics and planning, medical sciences, engineering and even commercial subjects such as marketing and branding. The two major pillars in this grand edifice are ‘survey designs’ and ‘evaluation designs’. In spite of the significant potential of statistical designs in the development of valid and reliable information, there is a wide and visible gap between development of theory and its practice.

Professor Leslie Kish, one of the world's top survey statisticians, once commented in his paper ‘The Hundred Years’ Wars of Survey Sampling’, “my central complaint is that over 95% of statistical attention in academia, textbooks and publications is devoted to mathematical statistical analysis and only 2% to design” (Kish, 2003a). With regard to attention to designs, his comment is as apt as ever. This is a rather unfortunate situation, particularly since the two designs can compliment and nurture each other's application and growth. Professor Kish continued to remark “the consequences of that neglect are too often poor designs by non-statisticians (engineers, economists etc.)”. Given this, the present book is a modest attempt to provide practitioners with tools for application of both designs, irrespective of their field of expertise.

A BRIEF HISTORY OF TWO DESIGNS

Understanding the prevalence of variables and their causal relationships has long been the prime focus of research. Initially, the interest was on obtaining a count of the population, such as adult population for tax purposes as well as for enrolment in the military, availability of land for habitation and agriculture etc. Graunt (1662) estimated the population of London around the year 1662. Utilising parish registers, he first estimated the ratio of number of burials per family. In fact, he observed that there would be 3 burials per 11 families in a year. He also obtained the total number of burials in London. With the help of the total and the ratio, he then estimated the total number of families. An assumption about the average family size then resulted in the estimated total population of the country (Bethlehem, 2009). Laplace used a type of survey to estimate the population of France in around the year 1812.