Installing mods in terraria requires the "TmodLoader" variant.
If you don't use this variant, you will need to :
- Delete all your server files
- Change the variant to "TModLoader" in the "variants" page
- Go back on the "manage" page and click "reinstall"
Once done, you will need to upload your mods, you can find how to do so in this article :
How to create a free terraria server.
Just navigate to the "Add mods with TmodLoader" part and learn how to mod your terraria server with TmodLoader.
If you ever need to edit the configuration settings of your mods, check the tutorial on how to edit a mod config for a terraria server.