From what I understand you obfuscate the port in order to limit the amount of incoming attacks. But then fail2ban would be a much more effective tool.
The disinterested aspect you described is the actual problem. Because it’s based on the assumption your port won’t be found, but it definitely will, and as soon as that happens you’ll end up in a database such as shodan and the entire effect is GONE.
May I interest you in firefly III?