The panel allocates empty space for items that are hidden by setting their Visibility to Collapsed. Such items should be ignored when the panel arranges its items.

To work this around, instead of the Visibility property, you can remove and add the items in the ItemSource collection whenever you need to toggle their visibility.