Can I terminate my lease because I'm pretty sure my landlord is abusing his access rights?

You signed a lease, what did it say? If it says he has open access, then you can't do anything. However, if it stipulates advance notice, then you may be able to bail.