I used to work in a shoe recycling shop...

I used to work in a shoe recycling shop. It was sole destroying.

This is a pun. The critical element of the pun is the word sole. A sole is the bottom of a shoe, but it also sounds like the word "soul", and even though I don't recall hearing the phrase "soul-destroying" very often, it does seem like something that would be said about their job by someone that didn't like it.