There are many anti soy articles online but none of them seem to provide any references to scientific articles that prove these claims to be true. There has been no conclusive evidence to date that shows a correlation between the consumption of soy products and the development of certain health issues to be true. Like any food I don't think that consuming excess amounts of soy would constitute a good diet, but eating in moderation along with a wide variety of fruits, vegetables and other wholefoods is fine.
Soy grown in Australia is GM free, and a lot of soy products will even have a GM soy free logo on the packaging. It's not expensive at all, Sanitarium soy milk is GM free and you can buy it from under $2 a litre from Woolworths and Coles.
Really you don't have anything to worry about, the amount of soy you eat sounds like a very healthy amount and unless you start eating it by the bucket load then you have nothing to worry about

This article was an interesting read:
http://zenhabits.net/soy/
Also take everything you read on the internet with a grain of salt, because you can find articles to prove or disprove just about anything, but it doesn't mean that their information is from a reliable source.