Investing all or most of your assets in a single property is not particularly safe [1]. You are exposed to a number of risks: you may go for some period without a tenant, the real estate agent managing the property may screw up or go feral, tenants may damage the property, rezoning or construction may affect its value, the suburb may become unfashionable, and so forth.
Property prices are smoothed out by the low frequency with which they change hands. If individual properties were valued every year, falls of 10% or even 20% might not be uncommon.
Direct property has high transaction costs and low granularity: it's not possible to invest just $10,000, or to extract $10,000 from a property without borrowing or some kind of financial instrument.
Also, individual properties are now so expensive that buying one may involve huge amounts of borrowing.
This "all eggs in one basket" problem gets even worse when, because property is so expensive, people join up with friends or family to buy an investment property together. Now it's "all your eggs in half a basket", with all kinds of possibilities for complications when one partner wants to sell or renovate and the other doesn't, or there are disagreements about management.
The returns on property can be good — anyone in Sydney holding property between 1999 and 2003 did pretty well — but are often not nearly as good as they appear at first sight.
The stories about people buying a house for $350,000 and selling it for $700,000 five years later often leave out, for the benefit of a good story, the $100,000 and 500 hours work that was put into renovations, not to mention the transaction costs. That might drop the return in this example from an apparent 15% pa to a still respectable 7% pa.
Property management is always going to take some time, even if you hand it over to a real estate agent.
The people who bought a property for $500,000 and had to sell it six years later for $500,000 — with a real loss even after rental returns, once inflation and transaction costs are taken into account — are unlikely to broadcast their stories so widely.
If you can buy property cheaply enough, in an area where prices will rise rapidly, it will be a good investment. But in a speculative boom there may be no bargains to be had, and doing the research to be able to find them is not so easy.
Having experience with owning or renting oneself is not the same thing as having knowledge of housing markets.
I'm not qualified to give financial advice and I know nothing about your personal circumstances. These are just my general thoughts.
Also, much of this page is specific to Australia, or even to New South Wales.
Some news stories:
Investors buying residential property have to compete with owner-occupiers.
The "income" to an owner-occupier from a property exceeds the rent to an investor from the same property, because of the premium accompanying ownership. (Which can partly be attributed to the low status and poor legal protection afforded renters.) Owner-occupiers are also exempt from land tax, and don't face some of the management costs/risks of an investor.
Owner-occupiers also get complete exemption from capital gains tax, instead of the 50% exemption of an investor holding for 12 months. So a typical owner-occupier will make a 15% to 23% higher after-tax capital gain than an investor buying and selling the same property.
Investors get tax deductibility on their loans, which owner-occupiers don't. But borrowing for any investment is tax deductible, and with some (for example forestry-MIS) the entire amount and not just the interest is tax-deductible.
The result is that owner-occupiers push the price of residential properties, especially houses, so high that they no longer make sense for investors.
This is what would happen in normal circumstances. In speculative booms, prices are set by people betting on unsustainable growth in house prices.
You can borrow more money at lower rates using property as security. (But gearing increases risk.)
Property provides a natural hedge against the need for somewhere to live. (But owning the property you live in offers this.)
Property provides a hedge against inflation. (But so do several other asset classes.)
Direct property can provide diversification to an investment portfolio. (But you'd need to be a multi-millionaire before even a cheap Sydney property made sense from this perspective.)
It is possible to increase the value of property through renovation or extension. (But are you investing or running a small construction business?)
As further evidence for my basic argument, there's a notable dearth of property trusts in Australia that hold ordinary residential property rather than commercial, retail or industrial property. Those few that do exist (for example Fortuna's) launched in 2006 and have no track record as yet. (And Westpac's "Residental Property Trust" turns out to own housing leased to the Defence forces on terms of around 10 years.)