simpson paradox

Simpson's paradox (or the Yule-Simpson effect) is a statistical paradox wherein the successes of groups seem reversed when the groups are combined. This result is often encountered in social and medical science statistics,^[1] and occurs when frequency data are hastily given causal interpretation;^[2] the paradox disappears when causal relations are derived systematically, through formal analysis.

Batting averages

A common example of the paradox involves batting averages in baseball: it is possible for one player to hit for a higher batting average than another player during a given year, and to do so again during the next year, but to have a lower batting average when the two years are combined. This phenomenon, which occurs when there are large differences in the number of at-bats between years, is well-known among sports sabermetricians such as Bill James.

A real-life example is provided by Ken Ross^[12] and involves the batting average of baseball players Derek Jeter and David Justice during the years 1995 and 1996:^[13]

	1995		1996		Combined
Derek Jeter	12/48	.250	183/582	.314	195/630	.310
David Justice	104/411	.253	45/140	.321	149/551	.270

In both 1995 and 1996, Justice had a higher batting average (in bold) than Jeter; however, when the two years are combined, Jeter shows a higher batting average than Justice. According to Ross, this phenomenon would be observed about once per year among the interesting baseball players. In this particular case, the paradox can still be observed if the year 1997 is also taken into account:

	1995		1996		1997		Combined
Derek Jeter	12/48	.250	183/582	.314	190/654	.291	385/1284	.300
David Justice	104/411	.253	45/140	.321	163/495	.329	312/1046	.298

[edit] Kidney stone treatment

This is a real-life example from a medical study^[14] comparing the success rates of two treatments for kidney stones.^[15]

The first table shows the overall success rates and numbers of treatments for both treatments (where Treatment A includes all open procedures and Treatment B is percutaneous nephrolithotomy):

Treatment A	Treatment B
78% (273/350)	83% (289/350)

This seems to show treatment B is more effective. If we include data about kidney stone size, however, the same set of treatments reveals a different answer:

	Treatment A	Treatment B
Small Stones	Group 1 93% (81/87)	Group 2 87% (234/270)
Large Stones	Group 3 73% (192/263)	Group 4 69% (55/80)
Both	78% (273/350)	83% (289/350)

The information about stone size has reversed our conclusion about the effectiveness of each treatment. Now treatment A is seen to be more effective in both cases. In this example the lurking variable (or confounding variable) of stone size was not previously known to be important until its effects were included.

Which treatment is considered better is determined by an inequality between two ratios (successes/total). The reversal of the inequality between the ratios, which creates Simpson's paradox, happens because two effects occur together:

1. The sizes of the groups, which are combined when the lurking variable is ignored, are very different. Doctors tend to give the severe cases (large stones) the better treatment (A), and the milder cases (small stones) the inferior treatment (B). Therefore, the totals are dominated by groups 3 and 2, and not by the two much smaller groups 1 and 4.
2. The lurking variable has a large effect on the ratios, i.e. the success rate is more strongly influenced by the severity of the case than by the choice of treatment. Therefore, the group of patients with large stones using treatment A (group 3) does worse than the group with small stones, even if the latter used the inferior treatment B (group 2).

[edit] Berkeley sex bias case

One of the best known real life examples of Simpson's paradox occurred when the University of California, Berkeley was sued for bias against women applying to graduate school. The admission figures for fall 1973 showed that men applying were more likely than women to be admitted, and the difference was so large that it was unlikely to be due to chance.^[16][3]

	Applicants	% admitted
Men	8442	44%
Women	4321	35%

However when examining the individual departments, it was found that no department was significantly biased against women; in fact, most departments had a small bias against men.

Major	Men		Women
	Applicants	% admitted	Applicants	% admitted
A	825	62%	108	82%
B	560	63%	25	68%
C	325	37%	593	34%
D	417	33%	375	35%
E	191	28%	393	24%
F	272	6%	341	7%

The explanation turned out to be that women tended to apply to competitive departments with low rates of admission even among qualified applicants (such as English), while men tended to apply to less-competitive departments with high rates of admission among qualified applicants (such as engineering). The conditions under which department-specific frequency data constitute a proper defense against charges of discrimination are formulated in Pearl (2000).

[edit] 2006 US school study

In July 2006, the United States Department of Education released a study^[17] documenting student performances in reading and math in different school settings.^[18] It reported that while the math and reading levels for students at grades 4 and 8 were uniformly higher in private/parochial schools than in public schools, repeating the comparisons on demographic subgroups showed much smaller differences, which were nearly equally divided in direction.

[edit] Low birth weight paradox

Main article: Low birth weight paradox

The low birth weight paradox is an apparently paradoxical observation relating to the birth weights and mortality of children born to tobacco smoking mothers. Traditionally, babies weighing less than a certain amount (which varies between countries) have been classified as having low birth weight. In a given population, low birth weight babies have a significantly higher mortality rate than others. However, it has been observed that low birth weight children born to smoking mothers have a lower mortality rate than the low birth weight children of non-smokers.^[19]

Vector interpretation

Simpson's paradox can also be illustrated using the 2-dimensional vector space.^[21] A success rate of p / q can be represented by a vector , with a slope of p / q. If two rates p₁ / q₁ and p₂ / q₂ are combined, as in the examples given above, the result can be represented by the sum of the vectors (q₁,p₁) and (q₂,p₂), which according to the parallelogram rule is the vector (q₁ + q₂,p₁ + p₂), with slope .

Simpson's paradox says that even if a vector (in blue in the figure) has a smaller slope than another vector (in red), and has a smaller slope than , the sum of the two vectors (indicated by "+" in the figure) can still have a larger slope than the sum of the two vectors , as shown in the example.

decoy effect

In marketing, the decoy effect (or asymmetric dominance effect) is the phenomenon whereby consumers will tend to have a specific change in preference between two options when also presented with a third option that is asymmetrically dominated. An option is asymmetrically dominated when it is inferior in all respects to one option; but, in comparison to the other option, it is inferior in some respects and superior in others. In other words, in terms of specific attributes determining preferability, it is completely dominated by (i.e., inferior to) one option and only partially dominated by the other. When the asymmetrically dominated option is present, a higher percentage of consumers will prefer the dominating option than when the asymmetrically dominated option is absent. The asymmetrically dominated option is therefore a decoy serving to increase preference for the dominating option. The decoy effect is also an example of the violation of the independence of irrelevant alternatives axiom of decision theory.

Example

For example, if there is a consideration set involving MP3 players, consumers will generally see higher storage capacity (number of GB) and lower price as positive attributes; while some consumers may want a player that can store more songs, other consumers will want a player that costs less. In Consideration Set 1, two devices are available:

Consideration Set 1

	A	B
price	$400	$300
storage	30GB	15GB

In this case, some consumers will prefer A for its greater storage capacity, while others will prefer B for its lower price.

Now suppose that a new player, C, is added to the market; it is more expensive than both A and B and has more storage than B but less than A:

Consideration Set 2

	A	B	C
price	$400	$300	$450
storage	30GB	15GB	25GB

The addition of C—which consumers would presumably avoid, given that a lower price can be paid for a model with more storage—causes A, the non-dominated option, to be chosen more often than if only the two choices in Consideration Set 1 existed; C affects consumer preferences by acting as a basis of comparison for A and B. Because A is better than C in both respects, while B is only partially better than C, more consumers will prefer A now than did before. C is therefore a decoy whose sole purpose is to increase sales of A.

Conversely, suppose that instead of C, a player D is introduced that has less storage than both A and B, and that is more expensive than B but not as expensive as A:

Consideration Set 3

	A	B	D
price	$400	$300	$350
storage	30GB	15GB	10GB

The result here is similar: consumers will not prefer D, because it is not as good as B in any respect. However, whereas C increased preference for A, D has the opposite effect, increasing preference for B.

things to do

cut hair...done
Find way to sell shades to make space for new one...on the way
Plan Vietnam trip...on the way
massage...done
paddle...on going
Gym...on going
run...on going
Do HT...resumed
Meet Prof before he goes for Christmas break...soon
Refurnish room...decide to postpone
do balance sheet...soon
watch movies...M2 and 4xmas done...

more

Further, what is the probability that

When there's no photo finish facility and they hired girls doing part-time for them to time the race?
Worse of all these people were sitting at a tower not aligned with the finishing line?
Why the fuck were they aligned with the half way mark for?

What is the probability that whenever its that old man who bleaches his hair, looks, dresses like a pedophile as my officer at the line-up.
My boat is never ask to 1 stroke up but 1 stroke back AND with my opponents 1 stroke up.
With the icing that it's not an one off incident but has happen for the past 3.5 years

Fucking pedophile: if your kayaking team can't win honorably against my coach's.
Please don't use such cheap tactics against his other DB teams to make your cock feel better.

Fairness? In this world to talk about fairness is to talk about money ah
Justice Bao died a long time liaoz

probability

What is the probability that NUS

Can have both boats in lane1 for their heats events?
while SIM can have lane 5 for both their mens and ivp heats?

What is the probability that on Saturday night SDBA uploaded the race schedule and line-up for Sunday, and NUS was to be in a semis that didn't seem so tough on paper
And then on Sunday morning when they reported the usher just on the spot says:
"hey everything is changed, line-up and event schedule. Oh you guys are NUS?
Ok Lane 1 for you"

what is the probability that the usher just keep scolding all the boats to buckle up their life-vests and to have footwear, if not they will disqualify the team...what is the probability that he's more afraid of someone falling off and drowning which will cause him lots of trouble versus he's really concerned of the safety of the paddlers?
If he's really concerned of the paddlers he wouldn't be using DQ as a threat to get the paddlers to behave
And why would he be shouting all those things to the paddlers only after they finish the race?

What is the probability that my banks heats was actually a photo finish between them and the second place, and when the results came out my team was actually third, with a timing that's 16 seconds behind the second place? And only after paying a upfront appeal fee of $150 then the thing was changed and my timing was cut by 16 seconds?

What is the probability that SDBA...a local ruling body for Singapore dragonboat, actually didn't have photo finish facility for the race...and everything was based on human timers with manual stopwatches for the lanes

What is the probability that at the public service race, the MOE team, compromising of all national paddlers...went off course....and demanded a re-race...and not willing to return back to shore....
such is the standard of moulders of the future of the nation
Can you imagine your child being taught by someone who spent 7 years getting a degree?
You know what he can teach your child? endurance

You know what is the probability of all the above mentioned happening?
Its the probability of seeing quantum teleportation if it's normal human beings, which by mathematical probability has only happened once since the birth of our universe.
If we are talking about SDBA, a body full of people who are interested in sucking money into their pockets, who prefer to spend money eating banquets themselves rather then building facilities for the people, who simply rest on their laurels because they are the single ruling body here...a peseudo monopoly,
then the probability is one
You can even see how the SDBA selection team's character being moulded and inherited from them man

left out on previous post

Forgot to say that the author has also mastered the proof that for a k-regular graph.
K will be its largest eigenvalue with multiplicity 1 and all one vector as its eigenvector

progress

First time was spent on getting the basic notations, definitions on algebraic graph theory

Next up the author learn/understand/write the three different kinds of proof for the classic Hoffman bound
The proofs uses linear algebra, semi definiteness and interlacing of eigenvalues respectively

Next up he considered a graph generated from a combinatorial point of view, claim is that the largest independent set of this partition graph is an equality on Hoffman
more precisely its an equality for the smallest eigenvalue

the largest independent set and clique has been found using Erdos-Ko -Rado theorem and "Berrynice" theorem...inferring that this graph satisfies AlphaG.OmegaG <= V(G) at equality

the author tried to classify this partition graph under one of the few known regular graph that satisfy hoffman at equality but failed

So the attempt to prove the conjecture begins,
People from another universoty has suggested using association schemes, generalize the phenomanon to known group theory results and project it down again
The author is currently looking at the matrices and main graph and its equitable partition.
There seems a pattern with the "compression matrix" containing the largest and smallest lamda of its "elder sibling"
The author conjectures this more general fact and hopes to prove it, which by far will then be a result in matrix theory and yet serves to prove his original conjecture on combinatorial graph theory

Currently the author is trying to come up with random sample matrices for his compression theory and using MATLAB, the conjecture has not broken down yet.