The Role of Statistical Methods in Large-Scale Data Analysis within Data Science: A Theoretical and Applied Study

Abstract

This research aims to highlight the role of statistical methods in the analysis of large-scale data within the framework of modern data science. With rapid advances in technologies for collecting, storing, and processing data, big data has become one of the most prominent challenges and opportunities for researchers and practitioners. The study adopts a mixed approach that combines a theoretical review of the literature with an applied analysis of the 1994 U.S. Adult Census Income dataset, using its training set of 32,561 observations and 14 demographic and economic variables. It should be noted that this dataset does not constitute big data in the full sense of the term; rather, it is a relatively large benchmark dataset selected as an applied example to illustrate the use of statistical methods and the issues that arise with large samples. The theoretical part reviews the concept of big data and its characteristics (Volume, Velocity, Variety, Veracity, and Value), together with the main statistical methods used in the field, including descriptive, inferential, and predictive approaches. The applied part presents a statistical analysis comprising descriptive statistics, the independent-samples t-test for comparing means, the chi-square test of independence for categorical variables, and correlation analysis, supplemented by effect size measures (Cohen’s d and Cramer's V). At a significance level of α = 0.05, the results showed statistically significant differences (p < 0.001) in mean age and weekly working hours between the two income groups, with medium effect sizes, as well as a statistically significant association between education level and income with a medium effect size (Cramer's V = 0.37), whereas the correlations among the quantitative variables were weak. The study concludes that statistical methods are a fundamental pillar of data science, and that applying them correctly, with attention to effect size rather than statistical significance alone, improves the accuracy of analysis and the reliability of data-driven decisions.