Math 151 , Day 39,  Wednesday, Nov. 28, 2007 .After class. Hit reload .

Exams returned.  Comments   Solutions         
Buffer against one low hour exam:
The final % exam grade minus 10 points will be substituted for the lowest hour exam grade, if it is higher.

This is to encourage  all to try to put it together for the (cumulative!) final.

Ch. 15: "Significance tests use an elaborate vocabulary, but the basic idea is simple: an outcome that would "rarely" happen if a claim were true--is good evidence that the claim is NOT true." (p.363 top)
I'm not making it up that this idea is important:  Financial Times (influential and high-end British newspaper) last weekend:
   (with formatting and pictures)  (without)   Statistical Significance: #10 of "The Ten Things Everyone Should Know About Science"

HW questions?  Day 38
Show real shoebox results.

Day37 for other details.  Summary, comments:

Take data.  Calculate test statistic. For µ, test statistic is the z-score of xbar. (Start with xbar, standardize using mean of H₀)
    Is it an unlikely result if  H₀ is true?  Then that is evidence against H₀.

Measuring the strength of the evidence against H₀ (a common measuring stick for all distributions and parameters):
P-value of a test:  The probability, computed assuming that H₀ is true, that the observed outcome would take a value as extreme or more extreme than that actually observed (if we could repeat taking-data again).  p. 368.
    The smaller the P-value, the stronger the data's evidence against H₀ ( for H_a).

For a test of µ  , using xbar (sigma known), the P-value is
--the area of the tail beyond the observed xbar, in the direction of H_a (one-sided)
(--or twice that area (two-sided).)
<>Applet:  P-value of a test of significance automates this.  (Uses "raw" scale of xbars, rather than z-scores). 

So for a test of a mean, the P-value for one-sided is half that for two sided, IF the result is in the direction of evidence for the alternative.

 "Significance level" details Day 38           
A "Significance level" alpha is a probability level we decide on  in advance as being the "rarely" amount that will push us over into believing (well, sort of) that the H₀ claim  is not true. (Historically older language than P-value.  Appropriate levels vary by discipline.)
We tend to use simple benchmark numbers for it, like .10 (1 in 10), .05 (1 in 20), .01 (1 in 100).
When the P-value is less  than (or equal to) a particular significance level alpha (say .05), we say,
    "The results are significant at the alpha = .05 level," or "The results are significant (P< .05)" .  Giving actual P is better, if you can.
Applet:  Applet: Statistical Significance   You can pick the alpha you desire, and see if your x-bar lies outside the "alpha" barrier(s). (approach of p. 376-79) But P-value is more informative.  (Some people  use "P-value" and "Significance level" to mean the same thing--P-value.)

What if you don't have the Z-table but only have the t-table (Table C)?   See Day 38 for details
What if you have a demanded level of significance, alpha?
    Table C: a limited list of probabilities  across the bottom rows:
            = Tail values for the bell curve distribution.   (one sided = one tail, two sided = two symmetrical tails)
        The value in the z* row above  P is the corresponding standard normal value ("critical value"). 
                 Check z* = 1.960, .025 above it (or below -1.960).  .05 farther out than it.  Corresponds to Table A.
      
  Do this: Find your z from the data. Make a sketch of the normal curve and mark your z on it.  Mark the direction(s) of H_a.
    (If your z is in the direction(s) of H_a, continue.  Otherwise the results are hopelessly not significant: you can quit.)
Find the two z*'s in Table C that bracket your z (ignore minus sign).  Find the corresponding P's.
    e.g. z =2.111
                                                 z = 2.111
      z*         2.054 \/ 2.326
One-sided P  ...  .02     .01
Two-sided P  ...  .04     .02 

      So the P-value for your z is: between .02 and .01 (If it's a one sided test)
         &  between double those 2 p's--between .04 and .02 (If it's a two sided test)
    Test is significant at the bigger bracketing probability; not sig. at the smaller.
If you have a specific demanded significance level, compare it with these levels.
            If  a test is significant at level b, then it is significant at every level bigger than b.
            If a test is Not significant at level d, then it is Not significant at every level smaller than d.
    "Significant at a":  probability of getting my results (again) by chance (if H₀ is true) is less than (or =) a. My result is less common than a.

- - - - - - - More NEW STUFF- -Do HW on this: I'll discuss Friday- -  - -
back to Ch. 16, cautions:  (SRS, normal pop., sigma known)
>>How small a P is convincing evidence against H₀?  (What alpha, to "Reject H₀?)
   --Is H_a surprising?  (Entrenched opinion is "for" H₀ .  )  Need strong evidence (small P).
   --Is rejecting  H₀ expensive?  Need strong evidence for H_a. 
        [May need to repeat experiment for doubters]
No sharp border between "significant" and "not"--though decisions may need to be made.

>>Statistical significance is not the same as practical significance ("clinical significance") 
      Tiny difference (from null value) can be statistically significant if sample size is large.
       Big difference may not be statistically significant if sample size is too small.
Do confidence intervals:  Estimate the size of the effect, not just yes/no of test..

Postpone the last: I'll do Friday.
>>Multiple Tests: beware! pp. 395-6
    If you do 100 tests and use the alpha = .05 significance level for each, then the structure of testing requires this:
    When all 100 null hypotheses H₀ are true, out of your 100, about 5 of the 100 (.05) will give "significant" results by chance alone (falsely indicating the alternative hypothesis is to be preferred.)   (10%--2 or 3-- of your 25 simulations of the shoebox with mean 20 will give "significant" (P< .10) results even though the mean is the null value of 20)  (My results)
Real shoeboxes (p.2) this term: only 1 falsely significant...
    Moral: if you use the testing mechanism as a screening instrument for many questions, a proportion will give falsely significant results.  You can't accept the results from such multiple tests as good evidence, only as indicating questions requiring further, more specific study. The game gives you one shot, not a hundred shots.    (This is becoming an important issue for developing new statistical techniques, for instance in biology, where microarrays can do a thousand tests at once.)
(not in text) You cannot legitimately test a hypothesis on the same data that first suggested that hypothesis. Every data set will turn up with some unusual pattern if you examine it hard enough. 
       (If you must explore and confirm with the same data set, one way is to (randomly) take half the data set, explore and generate hypotheses; then use the other half for confirmatory tests.  You can use P-value to describe unusualness, but be wary of making decisions with it if you didn't expect that particular unusualness.)

>>All the warnings about designing experiments and surveys still apply!