Preheat the oven to 350°F (175°C). Grease and flour a 9x5-inch loaf pan.
In a medium bowl, whisk together the flour, baking powder, baking soda, salt, cinnamon, and nutmeg. Set aside.
In a large mixing bowl, cream together the granulated sugar, brown sugar, and softened butter until light and fluffy.
Beat in the eggs one at a time, then stir in the vanilla extract.
Gradually mix in the dry ingredients alternately with the buttermilk, starting and ending with the dry ingredients. Mix until just combined.
Gently fold in the diced apples until evenly distributed.
Pour the batter into the prepared loaf pan and smooth the top with a spatula.
Bake in the preheated oven for 55-65 minutes, or until a toothpick inserted into the center comes out clean.
Allow the loaf to cool in the pan for 10 minutes before transferring it to a wire rack to cool completely.
For the glaze, whisk together the powdered sugar and milk in a small bowl until smooth. Drizzle over the cooled loaf cake before serving.