Cats and pregnant woman?

I am 17 weeks pregnant and although I don't have my own cat, there are lots of stray cats near my house which I feed and sometimes pet. I know that Toxo when ingested after contact with their faeces is dangerous to an unborn child. Obviously I have no contact with their pooh but I was wondering if its a problem my stroking them. I am always careful to wash my hands afterwards.