Installing mods in terraria requires the "TmodLoader" variant.
If you don't use this variant, you will need to :
- Delete all your server files
- Change the variant to "TModLoader" in the "variants" page
- Go back on the "manage" page and click "reinstall"
Once done, you will need to upload your mods, you can find how to do so in this article :
How to create a free terraria server.
Just navigate to the "Add mods with TmodLoader" part and learn how to mod your terraria server with TmodLoader.