Sex steroids during developmental period are critical to formation of sexually dimorphic nuclei in the brain, although the mechanisms of sex-steroid actions remain to be elucidated. We previously reported that masculinization of the principal nucleus of the bed nucleus of the stria terminalis (BNSTp), which is larger and has more neurons in male mice than in female mice, involves aromatized androgens that act via estrogen receptor-α (ERα), but not estrogen receptor-β (ERβ). Here, we examined sex steroid action on the formation of the anteroventral periventricular nucleus (AVPV) that exhibits sex differences opposite to those in BNSTp. Morphometrical analysis of aromatase knockout (AromKO), ERα knockout (αERKO), and ERβ knockout (βERKO) mice revealed that the volume and neuron number of the AVPV in AromKO and αERKO male mice were significantly smaller than those in wild-type males. We further examined the AVPV and BNSTp of androgen receptor knockout (ARKO) mice. The volume and neuron number of the BNSTp were significantly smaller in ARKO male mice than those in wild-type mice. These effects were not rescued by neonatal treatment of aromatizable testosterone. In contrast, there was no significant difference in the AVPV between ARKO and wild-type mice. We also investigated aromatase, ERα, ERβ, and AR mRNA levels in the AVPV and BNSTp of perinatal mice. The AVPV and BNSTp differed with regard to expression of each mRNA. These results suggested that androgen signaling via AR and estrogen signaling via ERα are requisite for masculinization of the BNSTp, whereas estrogen signaling via ERα is sufficient for defeminization of the AVPV.