[math] \require{textmacros} \def \bbeta {\bf \beta} \def\fat#1{\mbox{\boldmath$#1$}} \def\reminder#1{\marginpar{\rule[0pt]{1mm}{11pt}}\textbf{#1}} \def\SSigma{\bf \Sigma} \def\ttheta{\bf \theta} \def\aalpha{\bf \alpha} \def\ddelta{\bf \delta} \def\eeta{\bf \eta} \def\llambda{\bf \lambda} \def\ggamma{\bf \gamma} \def\nnu{\bf \nu} \def\vvarepsilon{\bf \varepsilon} \def\mmu{\bf \mu} \def\nnu{\bf \nu} \def\ttau{\bf \tau} \def\SSigma{\bf \Sigma} \def\TTheta{\bf \Theta} \def\XXi{\bf \Xi} \def\PPi{\bf \Pi} \def\GGamma{\bf \Gamma} \def\DDelta{\bf \Delta} \def\ssigma{\bf \sigma} \def\UUpsilon{\bf \Upsilon} \def\PPsi{\bf \Psi} \def\PPhi{\bf \Phi} \def\LLambda{\bf \Lambda} \def\OOmega{\bf \Omega} [/math]

Consider the logistic regression model to explain the variation of a binary random variable by a covariate. Let [math]Y_i \in \{ 0, 1\}[/math], [math]i=1, \ldots, n[/math], represent [math]n[/math] binary random variables that follow a Bernoulli distribution with parameter [math]P(Y_i = 1 \, | \, X_i) = \exp(X_i \beta) [1+ \exp(X_i \beta)][/math] and corresponding covariate [math]X_i \in \{ -1, 1 \}[/math]. Data [math]\{ (y_i, X_i) \}_{i=1}^n[/math] are summarized as a contingency table (Table):

	[math]y_i=\texttt{0}[/math]	[math]y_i=\texttt{1}[/math]
[math]X_i=\texttt{-1}[/math]	301	196
[math]X_i= \texttt{\textcolor{white}{-}1}[/math]	206	297

The data of Table is only to be used in parts [math]''c)''[/math] and [math]''g)''[/math].

• Show that the estimating equation of the ridge logistic regression estimator of [math]\bbeta[/math] is of the form:
  [[math]] \begin{eqnarray*} c - n \exp (\beta) [ 1 + \exp ( \beta)]^{-1} - \lambda \beta & = & 0, \end{eqnarray*} [[/math]]
  and specifiy [math]c[/math].
• Show that [math]\hat{\beta} (\lambda) \in (\lambda^{-1} (c-n), \lambda^{-1} c)[/math] for all [math]\lambda \gt 0[/math]. Ensure that this is a meaningful interval, i.e. that it is non-empty, and verify that [math]c \gt 0[/math]. Moreover, conclude from the interval that [math]\lim_{\lambda \rightarrow \infty} \hat{\beta} (\lambda) = 0[/math].
• The Taylor expansion of [math]\exp (\beta) [ 1 + \exp ( \beta)]^{-1}[/math] around [math]\beta=0[/math] is:
  [[math]] \begin{eqnarray*} \exp (\beta) [ 1 + \exp ( \beta)]^{-1} & = & \tfrac{1}{2} + \tfrac{1}{4} x - \tfrac{1}{48} x^3 + \tfrac{1}{480} x^5 + \mathcal{O} (x^6). \end{eqnarray*} [[/math]]
  Substitute the [math]3^{\mbox{{\tiny rd}}}[/math] order Taylor approximation into the estimating equations and use the data of Table to evaluate its roots using the polyroot-function (of the base-package). Do the same for the [math]1^{\mbox{{\tiny st}}}[/math] and [math]2^{\mbox{{\tiny nd}}}[/math] order Taylor approximations of [math]\exp (\beta) [ 1 + \exp ( \beta)]^{-1}[/math]. Compare these approximate estimates to the one provided by the ridgeGLM-function (of the porridge-package, ).

In the remainder consider the [math]1^{\mbox{{\tiny st}}}[/math] order Taylor approximated ridge logistic regression estimator: [math]\hat{\beta}_{1^{\mbox{{\tiny st}}}} ( \lambda ) = (\lambda + \tfrac{1}{4} n)^{-1} (c - \tfrac{1}{2} n)[/math].

• Find an expression for [math]\mathbb{E} [ \hat{\beta}_{1^{\mbox{{\tiny st}}}} (\lambda) ][/math].
• Find an expression for [math]\mbox{Var} [ \hat{\beta}_{1^{\mbox{{\tiny st}}}} (\lambda) ][/math].
• Combine the answers of parts d) and e) to find an expression for [math]\mbox{MSE} [ \hat{\beta}_{1^{\mbox{{\tiny st}}}} (\lambda) ][/math].
• Plot this MSE against [math]\lambda[/math] for the data provided in Table and reflect on analogue of Theorem for the ridge logistic regression estimator.