Gene-wise dispersion estimates To get a gene-wise dispersion estimate for a gene i, we start by fitting a negative binomial GLM without an LFC prior for the design matrix X to the gene’s count data. This GLM uses a rough method-of-moments estimate of dispersion, based on the within-group variances and means. The initial GLM is necessary to obtain an initial set of fitted values, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document} $\hat \mu _{\textit {ij}}^{0}$ \end{document}μ^ij0. We then maximize the Cox–Reid adjusted likelihood of the dispersion, conditioned on the fitted values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document} $\hat \mu _{\textit {ij}}^{0}$ \end{document}μ^ij0 from the initial fit, to obtain the gene-wise estimate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document} $\alpha ^{\text {gw}}_{i}$ \end{document}αigw, i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document} $$\alpha^{\text{gw}}_{i} = {\underset{\alpha}{\text{arg max}}}\; \ell_{\text{CR}}\left(\alpha; \vec\mu_{\textit{i}\cdot}^{0}, \vec K_{\textit{i}\cdot}\right) $$ \end{document}αigw=arg maxαℓCRα;μ→i·0,K→i· with (7)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document} $$ \begin{aligned} \ell_{\text{CR}}(\alpha; \vec\mu, \vec K) &= \ell(\alpha) - \frac{1}{2} \log\left(\det\left(X^{t} W X\right) \right) \\ \ell(\alpha) &= \sum\limits_{j} \log f_{\text{NB}}(K_{j}; \mu_{j}, \alpha), \end{aligned} $$ \end{document}ℓCR(α;μ→,K→)=ℓ(α)−12logdetXtWXℓ(α)=∑jlogfNB(Kj;μj,α),
