IP is like an address to a big skyscraper where a company operates. You are the delivery man and must go to 201.154.76.19 and deliver something. When you get at the reception, you tell them you have a package to deliver to Mrs HTTPS, at room (port) 443. Since Mrs HTTPS is well known and has cleared your entry before, you’re allowed to enter this room and only this room.
If you were to get at the same address and try to access other rooms you would either get refused because they are closed, or if open, someone would specifically need to be in the room so you can deliver something
Malicious actors that wanted access to the building could try to disguise their deliveries and enter the building, that’s why the default policy of most firewalls is “reject” and you specifically need to open a port and have a program listening to it if you want incoming connections.