Preheat the oven to 350°F (175°C). Grease a 9x5-inch loaf pan.
In a medium bowl, whisk together the flour, baking powder, baking soda, salt, cinnamon, and nutmeg. Set aside.
In a large bowl, cream together the granulated sugar, brown sugar, and softened butter until light and fluffy.
Beat in the eggs, one at a time, then stir in the vanilla extract.
Gradually add the dry ingredients to the wet mixture, alternating with the buttermilk. Mix until just combined.
Fold in the diced apples gently until evenly distributed.
Pour the batter into the prepared loaf pan and smooth the top.
Bake in the preheated oven for 50-60 minutes, or until a toothpick inserted into the center comes out clean.
Allow the loaf to cool in the pan for 10 minutes, then transfer to a wire rack to cool completely.
For the glaze, whisk together the powdered sugar, milk, and vanilla extract until smooth. Drizzle over the cooled loaf before serving.