Moving from broker-managed Mortgage to self-managed

When we purchased our flat, we decided to use the mortgage broker recommended by the property agency that sold it to us. However, we’re dissatisfied with their service and are considering managing the mortgage ourselves. Has anyone taken this approach? Anything to be aware of?