The only thing I can think of is this from the help documents:
Note:Topic monitoring uses the WebSocket communication protocol, which occurs over port 7078.If necessary, be sure to configure any firewall settings to allow for open communication on port 7078.
Are you using ARR in IIS?Does topic monitoring work when you login onto the server and try it there?
Hi
Are you maybe accessing FME Server over https?
If so, have a look here:https://knowledge.safe.com/questions/5151/notifications-seems-to-work-but-the-topic-monitor.html
David