Tainanfarm, I'm also looking into something similar. I'm just about to sign the contract for
renting 5 acres in Miaoli, which actually works out a lot cheaper (in terms of cost-of-capital) than buying, unless of course you're aiming to invest for your family. As someone else noted, farmland prices near cities are just daft. The reason is, people aren't selling farms, they're selling residential land. They know damn well it's being snapped up by building companies who will either (1) wait for rezoning (2) pressure the local bigwig for rezoning (3) flout the law and build anyway, usually taking advantage of the 1/10th-area "farm building" laws to put up a 5-storey house.
If you're interested in the East coast, however, that looks like a whole different story. I've seen land there going for NT$100 per ping (US$10K/ha), or even less, and I aim to get a slice as soon as my wife can learn to stop spending all my money

At that price, it's generally 'forest' land, but if you're familiar with the latest forest agriculture or permaculture techniques, that's not an issue even for commercial farming. In my experience western-style agriculture simply doesn't work in tropical locations (the land gets stripped and depleted even quicker than it does in temperate climates) and I have no idea why Taiwanese farmers persist with it.
I also have been told by several estate agents that it's fine for foreigners to buy land as long as your home country allows Taiwanese people to buy land. I've heard there is an upper size limit and limitations for commercial-sized purchases (several hectares) but I don't know the details. I doubt anybody will lend you money though. You have to be pureblood Han to get cash from Taiwanese banks.
As for cheap sheds/shacks ... I'm planning to put a small rammed-earth building on my rental plot. It only needs a poured-concrete foundation (which, in my case, is already there), some formwork, and lots of hard work. Rammed earth is earthquake-safe and attractive when done properly, and it's easily demolished if/when the contract runs out.