To be fair, I only agree with Toriyama to a certain extent. Goku isn't a complete disaster or a bad father per say. He definitely occupies that middle of the road for his sons. Now what people are forgetting is that he's an absolute friggin' train wreck as a husband. This is also putting our Chi-Chi disdain aside as well. He sucks as a partner and I have no idea how their marriage works.
Chi Chi sucks as a wife as well though, so I think that's why people always forget that Goku sucks as a husband. I also think that Piccolo was so much better of a father figure to Gohan than Goku was as well, so that's why people think Goku is a bad father, at least in comparison to Piccolo.