Make sure you're using a game version that is compatible with the mods you'd like to install, the 2 most common mod loaders are Fabric and Forge. Fabric mods do not work with Forge and vice versa.
As mods are version specific, they will only work on the version of the game they were created for and not any others. A mod created for Forge 1.16.5 will only work on Forge version 1.16.5. If you try to install it on a 1.20.4 server, it either won’t load at all or the server will crash on startup.
Installing plugins automatically
The simplest way is to add plugins via the Plugins panel in your instance manager. Simply search up the plugins you want to add, and select install. When you restart your server, it's recommended to keep the console open, in order to see if there are any issues with installing the plugin. The console will let you know if you are missing dependencies, or if any other issues happen.
Installing plugins or mods manually
Make sure you download the mod files for the correct version of Minecraft that you're running. Save these locally to your computer. Open the management panel, and enter File Manager. If you do not already have a plugins or mods folder, you can create them. Right click on the page, and select "Create Directory" to create the folder.
Make sure you enter the correct folder on the server, and then simply drag the file from your computer, to the file manager. It should then upload the files.
Once you have installed mods, it is recommended to watch the console as your server boots to check for any issues.