When you try to set the LabelDefinition property in style it is not applied. As a workaround you can create an attached property to which you can apply the custom definition and set it to the LabelDefinition property of the annotation in its OnPropertyChangedCallback.
Hi, On 22 October 2018 (Monday), we are uploading our next LIB version (2018.3.1022) where the fix will be included. It will also be included in our official version – R1 2019, scheduled for the mid of January 2019. Best regards, Ralitsa Kumanova