different statistical significance everytime odp() function is called

[sridewi1]

Hi,

I realized that every time I called the odp() function with the identical input and parameters, I got a totally different results for the statistical significance summary pi0 and cumulative number of significance calls. How is this possible?

here are two different summary(de_obp) of the results using the same command after calling the odp(de_obj, verbose=FALSE, n.mods=50, bs.its=100) consecutively

Statistical significance summary:
pi0:    0.1436803

Cumulative number of significant calls:

          <1e-04 <0.001 <0.01 <0.025 <0.05 <0.1  <1
p-value       87    156   263    304   337  368 495
q-value       92    195   336    384   418  471 495
local fdr     58    132   239    282   316  346 484

the second result:

Statistical significance summary:
pi0:    0.2391582

Cumulative number of significant calls:

          <1e-04 <0.001 <0.01 <0.025 <0.05 <0.1  <1
p-value      146    199   275    316   341  373 495
q-value      151    217   317    362   389  436 495
local fdr    121    174   228    266   294  326 448

thanks in advance for any kind of help.
Dewi

[ajbass]

The odp function runs a bootstrap algorithm to calculate the null distribution (to get p-values). It really depends on the number of tests (or genes) you have. Here there are not many tests, and so you would want to run a lot of bootstraps. I would recommend increasing bs.its:

odp(de_obj, verbose=FALSE, n.mods=50, bs.its = 1000) 

In the vignette we use a very small subset of the actual data to demonstrate how to use the commands.

[sridewi1]

Thank you for the prompt reply.
I did just as you suggested and go even higher to bs.its = 1000000, but I still got two different summaries. My dataset consists of 495 features in 35 different samples. How high should or can I go with the bootstrap iteration?

ExpressionSet Summary

ExpressionSet (storageMode: lockedEnvironment)
assayData: 495 features, 35 samples
  element names: exprs
protocolData: none
phenoData
  sampleNames: E14F1 E14F2 ... POM6 (35 total)
  varLabels: adj.var tme
  varMetadata: labelDescription
featureData: none
experimentData: use 'experimentData(object)'
Annotation:

de Analysis Summary

Total number of arrays: 35
Total number of probes: 495

Biological variables:
        Null Model:~adj.var
<environment: 0x56465c3c5748>

        Full Model:~adj.var + ns(tme, df = 2, intercept = FALSE)
<environment: 0x56465c3c5748>

.......


Statistical significance summary:
pi0:    0.1498209

Cumulative number of significant calls:

          <1e-04 <0.001 <0.01 <0.025 <0.05 <0.1  <1
p-value      134    200   282    311   345  375 495
q-value      152    245   342    383   428  464 495
local fdr     97    171   259    287   312  349 495

the second result using the same parameter bs.its=1000000

ExpressionSet Summary

ExpressionSet (storageMode: lockedEnvironment)
assayData: 495 features, 35 samples
  element names: exprs
protocolData: none
phenoData
  sampleNames: E14F1 E14F2 ... POM6 (35 total)
  varLabels: adj.var tme
  varMetadata: labelDescription
featureData: none
experimentData: use 'experimentData(object)'
Annotation:

de Analysis Summary

Total number of arrays: 35
Total number of probes: 495

Biological variables:
        Null Model:~adj.var
<environment: 0x56465c3c5748>

        Full Model:~adj.var + ns(tme, df = 2, intercept = FALSE)
<environment: 0x56465c3c5748>

.......


Statistical significance summary:
pi0:    0.1248155

Cumulative number of significant calls:

          <1e-04 <0.001 <0.01 <0.025 <0.05 <0.1  <1
p-value      133    198   282    322   345  371 495
q-value      157    249   353    402   442  479 495
local fdr    100    175   264    305   333  365 495