Statistics by @cosmog97

 An “analog” of the CLT for stochastic process: the standard Wiener process as “scaling limit” of a random walk and the functional CLT (Donsker theorem) or invariance principle. Explain the intuitive meaning of this result and how you have already illustrated the result in your homework

By clearly recalling the different idealized assumptions in this article, the underlying dynamics of the Brownian particle being knocked about by molecules suggests a random walk as a possible model, but with tiny time steps and tiny spatial jumps. Let X = (X₀, X₁, X₂, …) be the symmetric simple random walk. Thus, X_n =   where U = (U₁, U₂, …) is a sequence of independent variables with P(U_i = 1) = P(U_i = −1) = 1/2 for each i ∈ N+. Recall that E(X_n) = 0 and var(X_n) = n for n ∈ N. Also, since X is the partial sum process associated with an IID sequence, X has stationary, independent increments (but of course in discrete time). Finally, recall that by the central limit theorem, converges to the standard normal distribution as n → ∞. Now,  for h,d ∈ (0,∞) the continuous time process

is a jump process with jumps at {0, h, 2h, …} and with jumps of size ±d. Basically, we would like to let h↓0 and d↓0, but this cannot be done arbitrarily. Note that E[X_h,d(t)] = 0 but var[X_h,d(t)] = d²⌊t/h⌋. Thus, by the central limit theorem, if we take d = then the distribution of X_h,d(t) will converge to the normal distribution with mean 0 and variance t as h↓0. More generally, we might hope that all of the requirements in the definition are satisfied by the limiting process, and if so, we have a standard Brownian motion. [1][2]

REFERENCES

[1] stats.libretexts.org, “Standard Brownian Motion”, URL.
[2] en.wikipedia.org, “Brownian motion”, URL.

What is the “Brownian motion” and what is a Wiener process. History, importance, definition and applications (Bachelier, Wiener, Einstein, …)

In 1827, the botanist Robert Brown noticed that tiny particles from pollen, when suspended in water, exhibited continuous but very jittery and erratic motion. In his miracle year in 1905, Albert Einstein explained the behavior physically, showing that the particles were constantly being bombarded by the molecules of the water, and thus helping to firmly establish the atomic theory of matter. Brownian motion as a mathematical random process was first constructed in a rigorous way by Norbert Wiener in a series of papers starting in 1918. For this reason, the Brownian motion process is also known as the Wiener process.

Along with the Bernoulli trials process and the Poisson process, the Brownian motion process is of central importance in probability. Each of these processes is based on a set of idealized assumptions that lead to a rich mathematical theory. In each case also, the process is used as a building block for a number of related random processes that are of great importance in a variety of applications. In particular, Brownian motion and related processes are used in applications ranging from physics to statistics to economics.

The French mathematician Louis Bachelier in his doctoral thesis (1900) on the “Théorie de la spéculation” developed a theory, based on a statistical approach, with the aim of accounting for the trend in the prices of securities on the Paris Stock Exchange. The mathematical tools used by him are very similar to those used by Einstein in the analysis of Brownian motion, and they share the fundamental assumptions: that the variations of the quantity in question (the prices of the securities in this case, the displacements in that of the of the particles) are independent from the previous ones, and that the probability distribution of these variations is Gaussian.

For this work, which represents the first mathematical representation of the trend over time of economic-financial phenomena, Bachelier is considered the father of mathematical finance: in his honor the Croatian-American mathematician William Feller proposed to indicate the Wiener process as a process by Bachelier – Wiener.

Definition

A standard Brownian motion is a random process X = {Xt : t ∈ [0,∞)} with state space R that satisfies the following properties:

• X₀ = 0 (with probability 1).
• X has stationary increments. That is, for s, t ∈ [0,∞) with s < t, the distribution of X_t−X_s is the same as the distribution of X_t−s.
• X has independent increments. That is, for t₁, t₂, …, t_n ∈ [0,∞) with t₁ < t₂ < ⋯ < t_n, the random variables X_t1, X_t2−X_t1, …, X_tn−X_tn−1 are independent.
• X_t is normally distributed with mean 0 and variance t for each t ∈ (0,∞).
• With probability 1, t ↦ X_t is continuous on [0,∞).

To understand the assumptions, let’s take them one at a time.

Suppose that we measure the position of a Brownian particle in one dimension, starting at an arbitrary time which we designate as t = 0, with the initial position designated as x = 0. Then this assumption is satisfied by convention. Indeed, occasionally, it’s convenient to relax this assumption and allow X₀ to have other values.

This is a statement of time homogeneity: the underlying dynamics (namely the jostling of the particle by the molecules of water) do not change over time, so the distribution of the displacement of the particle in a time interval [s,t] depends only on the length of the time interval. This is an idealized assumption that would hold approximately if the time intervals are large compared to the tiny times between collisions of the particle with the molecules.

This is another idealized assumption based on the central limit theorem (CLT): the position of the particle at time t is the result of a very large number of collisions, each making a very small contribution. The fact that the mean is 0 is a statement of spatial homogeneity: the particle is no more or less likely to be jostled to the right than to the left. Next, recall that the assumptions of stationary, independent increments means that var(X_t) = σ²t for some positive constant σ². By a change in time scale, we can assume σ² = 1. [1][2][3]

References

[1] stats.libretexts.org, “Standard Brownian Motion”, URL.
[2] en.wikipedia.org, “Brownian motion”, URL.
[3] Focus, “Che cosa è il moto browniano?”, URL

Do a research about the general correlation coefficient for ranks and the most common indices that can be derived by it. Do one example of computation of these correlation coefficients for ranks.

Correlation is a statistical method used to assess a possible linear association between two continuous variables. It is simple both to calculate and to interpret. However, misuse of correlation is so common among researchers that some statisticians have wished that the method had never been devised at all. [1]

A correlation coefficient is a number between -1 and 1 that tells you the strength and direction of a relationship between variables. In other words, it reflects how similar the measurements of two or more variables are across a dataset. Correlation coefficients are unit-free, which makes it possible to directly compare coefficients between studies.

The sign of the coefficient reflects whether the variables change in the same or opposite directions: a positive value means the variables change together in the same direction, while a negative value means they change together in opposite directions.

The absolute value of a number is equal to the number without its sign. The absolute value of a correlation coefficient tells you the magnitude of the correlation: the greater the absolute value, the stronger the correlation.

Before going on in reading, let’s remember that the correlation coefficient is a bivariate statistic when it summarizes the relationship between two variables, and it’s a multivariate statistic when you have more than two variables. It is also an effect size measure, which tells you the practical significance of a result.

Visualizing linear correlations

The correlation coefficient tells you how closely your data fit on a line. If you have a linear relationship, you’ll draw a straight line of best fit that takes all of your data points into account on a scatter plot. The closer your points are to this line, the higher the absolute value of the correlation coefficient and the stronger your linear correlation.

If all points are perfectly on this line, you have a perfect correlation:

If these points are spread far from this line, the absolute value of your correlation coefficient is low:

Note that the steepness or slope of the line isn’t related to the correlation coefficient value. The correlation coefficient doesn’t help you predict how much one variable will change based on a given change in the other because two datasets with the same correlation coefficient value can have lines with very different slopes.

Types of correlation coefficients

You can choose from many different correlation coefficients based on the linearity of the relationship, the level of measurement of your variables, and the distribution of your data.

Pearson’s r

The Pearson’s product-moment correlation coefficient, also known as Pearson’s r, describes the linear relationship between two quantitative variables.

These are the assumptions your data must meet if you want to use Pearson’s r:

• Both variables are on an interval or ratio level of measurement
• Data from both variables follow normal distributions
• Your data have no outliers
• Your data is from a random or representative sample
• You expect a linear relationship between the two variables

Pearson’s r is a parametric test, so it has high power. But it’s not a good measure of correlation if your variables have a non-linear relationship, or if your data have outliers, skewed distributions, or come from categorical variables. If any of these assumptions are violated, you should consider a rank correlation measure.

The formula for Pearson’s r is complicated, but most computer programs can quickly churn out the correlation coefficient from your data. In a simpler form, the formula divides the covariance between the variables by the product of their standard deviations.

with:

• r_xy= strength of the correlation between variables x and y
• n = sample size
• ∑ = sum of what follows…
• X = every x-variable value
• Y = every y-variable value
• XY = the product of each x-variable score and the corresponding y-variable score

When using the Pearson correlation coefficient formula, you’ll need to consider whether you’re dealing with data from a sample or the whole population. The sample and population formulas differ in their symbols and inputs. A sample correlation coefficient is called r, while a population correlation coefficient is called rho, the Greek letter ρ.

The sample correlation coefficient uses the sample covariance between variables and their sample standard deviations. The population correlation coefficient uses the population covariance between variables and their population standard deviations.

with:

• ρ_XY, r_xy  = strength of the correlation between variables x and y
• cov(X,Y), cov(x,y) = covariance of x and y
• s_x = sample standard deviation of x
• s_y = sample standard deviation of y
• σ_X = population standard deviation of X
• σ_Y = population standard deviation of Y

Spearman’s rho

Spearman’s rho, or Spearman’s rank correlation coefficient, is the most common alternative to Pearson’s r. It’s a rank correlation coefficient because it uses the rankings of data from each variable (e.g., from lowest to highest) rather than the raw data itself.

You should use Spearman’s rho when your data fail to meet the assumptions of Pearson’s r. This happens when at least one of your variables is on an ordinal level of measurement or when the data from one or both variables do not follow normal distributions.

While the Pearson correlation coefficient measures the linearity of relationships, the Spearman correlation coefficient measures the monotonicity of relationships. In a linear relationship, each variable changes in one direction at the same rate throughout the data range. In a monotonic relationship, each variable also always changes in only one direction but not necessarily at the same rate.

• Positive monotonic: when one variable increases, the other also increases.
• Negative monotonic: when one variable increases, the other decreases.
• Monotonic relationships are less restrictive than linear relationships.

The symbols for Spearman’s rho are ρ for the population coefficient and rs for the sample coefficient. The formula calculates the Pearson’s r correlation coefficient between the rankings of the variable data. To use this formula, you’ll first rank the data from each variable separately from low to high: every data point gets a rank from first, second, or third, etc. Then, you’ll find the differences (di) between the ranks of your variables for each data pair and take that as the main input for the formula.

with:

• r_s= strength of the rank correlation between variables
• d_i = the difference between the x-variable rank and the y-variable rank for each pair of data
• ∑d²_i = sum of the squared differences between x- and y-variable ranks
• n = sample size

If you have a correlation coefficient of 1, all of the rankings for each variable match up for every data pair. If you have a correlation coefficient of -1, the rankings for one variable are the exact opposite of the ranking of the other variable. A correlation coefficient near zero means that there’s no monotonic relationship between the variable rankings.

Other coefficients

When you square the correlation coefficient, you end up with the correlation of determination (r²). This is the proportion of common variance between the variables. The coefficient of determination is always between 0 and 1, and it’s often expressed as a percentage.

A high r² means that a large amount of variability in one variable is determined by its relationship to the other variable. A low r² means that only a small portion of the variability of one variable is explained by its relationship to the other variable; relationships with other variables are more likely to account for the variance in the variable.

When you take away the coefficient of determination from unity (one), you’ll get the coefficient of alienation. This is the proportion of common variance not shared between the variables, the unexplained variance between the variables.

A high coefficient of alienation indicates that the two variables share very little variance in common. A low coefficient of alienation means that a large amount of variance is accounted for by the relationship between the variables. [2]

An example of calculating Spearman’s correlation

To calculate a Spearman rank-order correlation on data without any ties we will use the following data:

We then complete the following table:

Where d = difference between ranks and d² = difference squared. We then calculate the following:

as n = 10. Hence, we have a ρ (or r_s) of 0.67. This indicates a strong positive relationship between the ranks individuals obtained in the Maths and English exams. That is, the higher you ranked in Maths, the higher you ranked in English also, and vice versa. [3][4]

REFERENCES

[1] MM Mukaka, “A guide to appropriate use of Correlation coefficient in medical research”, URL.
[2] Pritha Bhandari, “A guide to correlation coefficients”, URL.
[3] Lund Research, “Spearman’s Rank-Order Correlation”, URL.
[4] questionpro.com, “Spearman correlation coefficient: Definition, Formula and Calculation with Example“, URL.

Distributions of the order statistics: look on the web for the most simple (but still rigorous) and clear derivations of the distributions, explaining in your own words the methods used.

Order statistics are a very useful concept in statistical sciences. They have a wide range of applications including modeling auctions, car races, and insurance policies, optimizing production processes, estimating parameters of distributions, et al. Through this article, we’ll understand the idea of order statistics. We’ll first understand its meaning and gradually proceed to its distribution, eventually covering more advanced concepts.

Suppose we have a set of random variables X₁, X₂, …, X_n, which are independent and identically distributed (i.i.d). By independence, we mean that the value taken by a random variable is not influenced by the values taken by other random variables. By identical distribution, we mean that the probability density function (PDF) (or equivalently, the Cumulative distribution function, CDF) for the random variables is the same. The k^th order statistic for this set of random variables is defined as the k^th smallest value of the sample.

To better understand this concept, we’ll take 5 random variables X₁, X₂, X₃, X₄, X₅. We’ll observe a random realization/outcome from the distribution of each of these random variables. Suppose we get the following values:

The k^th order statistic for this experiment is the k^th smallest value from the set {4, 2, 7, 11, 5}. So, the 1^st order statistic is 2 (smallest value), the 2^nd order statistic is 4 (next smallest), and so on. The 5^th order statistic is the fifth smallest value (the largest value), which is 11. We repeat this process many times i.e., we draw samples from the distribution of each of these i.i.d random variables, & find the k^th smallest value for each set of observations. The probability distribution of these values gives the distribution of the k^th order statistics.

In general, if we arrange random variables X₁, X₂, …, X_n in ascending order, then the k^th order statistic is shown as:

The general notation of the k^th order statistic is X_(k). Note X_(k) is different from X_k. X_k is the k^th random variable from our set, whereas X_(k) is the k^th order statistic from our set. X_(k) takes the value of X_k if X_k is the k^th random variable when the realizations are arranged in ascending order.

The 1^st order statistic X₍₁₎ is the set of the minimum values from the realization of the set of ‘n’ random variables. The n^th order statistic X_(n) is the set of the maximum values (nth minimum values) from the realization of the set of ‘n’ random variables. They can be expressed as:

Distribution of Order Statistics

A) Distribution of the n^th Order Statistic:

Let the probability density function (PDF) & cumulative distribution function (CDF) our random variables be f_x(x), and F_x(x) respectively. By definition of CDF,

Since our random variables are identically distributed, they have the same PDF f_x(x) & CDF F_x(x). We’ll now calculate the CDF of n^th order statistic (F_n(x)) as follows:

The random variables X₁, X₂, …, X_n are also independent. Therefore, by property of independence,

 

The PDF of the n^th order statistic (f_n(x)) is calculated as follows:

Thus, the expression for the PDF & CDF of n^th order statistic has been obtained.

B) Distribution of the 1^st Order Statistic:

The CDF of a random variable can also be calculated as the one minus the probability that the random variable X takes a value more than or equal to x. Mathematically,

We’ll determine the CDF of 1^st order statistic (F₁(x)) as follows:

 

Once again, using the property of independence of random variables,

 

The PDF of the 1^st order statistic (f₁(x)) is calculated as follows:

Thus, the expression for PDF & CDF of 1^st order statistic has been obtained.

C) Distribution of the k^th Order Statistic:

For k^th order statistic, in general, the following equation describes its CDF (F_k(x)):

The PDF of k^th order statistic (f_k(x)) is expressed as:

To avoid confusion, we’ll use geometric proof to understand the equation. As discussed before, the set of random variables have the same PDF (f_X(x)). The following graph shows a sample PDF with the k^th order statistic obtained from random sampling:

So, the PDF of the random variables f_X(x) is defined between the interval [a,b]. The kth order statistic for a random sample is shown by the red line. The other variable realizations (for the random sample) are shown by the small black lines on the x-axis.

There are exactly (k – 1) random variable observations that fall in the yellow region of the graph (the region between a & k^th order statistic). The probability that a particular observation falls in this region is given by the CDF of the random variables (F_X(x)). But we are aware that (k – 1) observations did fall in the region, which gives us the term (by independence) (F_X(x))^{(k – 1)}.

There are exactly (n – k) random variable observations that fall in the blue region of the graph (the region between k^th order statistic & b). The probability that a particular observation falls in this region is given by the 1 – CDF of the random variables (1– F_X(x)). But we are aware that (n – k) observations did fall in the region, which gives us the term (by independence) (1–F_X(x))^{(n – k)}.

Finally, exactly 1 observation falls exactly at the kth order statistic with probability f_X(x). Thus, the product of the 3 terms gives us an idea of the geometric meaning of the equation for PDF of the kth order statistic. But where does the factorial term come from? The above scenario just showed one of the many orderings. There can be many such combinations. The total number of such combinations is shown as follows:

Thus, the product of all of these terms gives us the general distribution of the k^th order statistic.

Useful Functions of Order Statistics

Order statistics give rise to various useful functions. Among them, the notable ones include sample range and sample median.

1) Sample range: It is defined as the difference between the largest and smallest value. It is expressed as follows:

2) Sample median: The sample median divides the random sample (realizations from the set of random variables) into two halves, one that contains samples with lower values, and the other that contains the samples with higher values. It’s like the middle/central order statistic. It is mathematically defined as:

Joint PDF of Order Statistics

A joint probability density function can help us better understand the relationship between two random variables (two order statistics
in our case). The joint PDF for any 2 order statistics X_(a) & X_(b), such that 1 ≤ a ≤ b ≤ n is given by the following equation:

Example

We’ll use a very simple example to illustrate the distribution of order statistics- the standard uniform distribution (U[0, 1] distribution). We’ll take 5 random variables X₁, X₂, X₃, X₄, X₅, all having the U[0, 1] distribution. For this set of random variables, we’ll calculate & plot the 1^st, 3^rd (the sample median) & 5^th (n^th) order statistics. The following figure shows the U[0, 1] distribution:

We’ll draw random samples as follows and find the 1^st, 3^rd & 5^th order statistics for each sample. Two of the samples are shown below:

The PDF & CDF of standard uniform distribution is given as:

We’ll use this information and calculate X₍₁₎, X₍₃₎ & X₍₅₎ using the formulas we derived. We’ll take the case only when x is between 0 & 1 (for other cases, the order statistic is zero as PDF is zero). [1][2][3]

A) For 1^st order statistics:

B) For 3^rd order statistics:

C) For 5^th order statistics:

REFERENCES

[1] Naman Agrawal, “Introduction to Order Statistics“, URL.
[2] colorado.edu, “Order Statistics”, URL.
[3] statisticshowto.com, “Order Statistics: Simple Definition, Examples”, URL.