Preheat the oven to 350°F (175°C). Grease and flour a 9x5 inch loaf pan.
In a medium bowl, whisk together the flour, baking powder, baking soda, salt, cinnamon, and nutmeg. Set aside.
In a large bowl, cream together the granulated sugar, brown sugar, and softened butter until light and fluffy.
Beat in the eggs, one at a time, followed by the vanilla extract.
Gradually add the dry ingredients to the wet ingredients, alternating with the buttermilk. Mix until just combined.
Gently fold in the diced apples and chopped nuts, if using.
Pour the batter into the prepared loaf pan, smoothing the top.
Bake in the preheated oven for 55-65 minutes, or until a toothpick inserted into the center comes out clean.
Allow the loaf to cool in the pan for 10 minutes before transferring it to a wire rack to cool completely.