Loans can be expensive, but sometimes it's either sell your soul to the bank and fly a lot or don't fly enough, if at all.
Don't feel bad about moving on to another CFI. You'll probably have many. I've had about eight or so. I still keep in contact with most.