One thing I share with Ludwig Wittgenstein is a hostility towards metaphysics.
Henry Le Roy Finch argues* that the origin of metaphysics lies in the idea of identity, which he traces to Plato's conception of original or self-existing things, and to Aristotle's (and the Aristotelian tradition's) more systematically logical approach to the notion of self-identity.
That a thing is identical with itself (traditionally referred to as Aristotle's first law of thought) is often seen as the foundation for all logic.
Wittgenstein, on the other hand, thought that there was no more meaningless statement than a statement of self-identity. "To say of one thing that it is identical with itself is to say nothing at all." (Tractatus 5.5303)
But Finch goes further and suggests that this skepticism about self-identity is linked to Wittgenstein's rejection of the popular notion of personal identity, the Cartesian thinking self. This is a central theme of Wittgenstein's (subsequently taken up by Gilbert Ryle). As Wittgenstein put it: "There is no such thing as the subject that thinks or entertains ideas." (Tractatus 5.631)
Certainly, both this claim and the previously-cited one are anti-metaphysical. And, significantly, Wittgenstein saw metaphysics and religion (or at least the sort of religion he embraced) as being in opposition to one another rather than as allies.
Finch rightly points out that "identitylessness" is at the heart of some important religious traditions, notably Buddhism, certain forms of Christianity** and Islam in its Sufi aspect.
Wittgenstein said that his goal was to "show the fly the way out of the fly-bottle", which can be interpreted as referring to the freeing of the human being from his or her false self-perception as a thinking self in its own private world. And this view of freedom is quite consistent with the religious traditions listed above.
On the other hand, as Finch points out, it is not consistent with other religious and philosophical approaches:
"The Stoic (and some would say also Judaic) idea of freedom is essentially that of Kant, which is that of the ethical self or free will, in which the self still retains its identity through its capacity to decide."
I remain uncomfortable with religious language and concepts, but I don't think someone like Wittgenstein can be understood (and I think he is worth trying to understand) if one ignores the implicit religious dimensions of his thought.
Also, having grown up (and so having invested a lot) in a religious tradition which I subsequently rejected, it's satisfying to see elements of that tradition coming into play here in a positive way.
* See his Wittgenstein, published by Element Books as part of the series Masters of Philosophy.
** I would single out the tradition known as fideism, and also the various mystical traditions. Pauline themes are important here; and it is worth noting in this connection that Wittgenstein liked the writings of the twentieth-century theologian Karl Barth.