You should delete it.
Let's say you have ClassA and ClassB. And ClassA calls methods of ClassB. Then you delete ClassB from the repository, but it stays locally. And everything is fine for you and works, but down the line you start a new installation and suddenly it does not work, because ClassB does not exist.
When you're working with Git, or any version control system for that matter, you can easily rollback to any state of the repository. So once you commited something you can always get it back.
On some of our projects we first mark stuff we want to delete as DEPRECATED. But it stays in the codebase and in the repository. After a while, after we're reasonably sure that we in fact do not need this, the actual delete from the repository and from the server happens.