If anyone's interested - two friends and I once went to Taichung Harbour last year winter, and were thinking we could get to Penghu from there. Of course, there was nothing going there, and everyone kept telling us we should've flown there. In the end, we didn't get there, but it is interesting to note that the local operator of boat-tours offered to take us there for $6,000 only. Of course, that'd have been a one-way trip, but the boat could've held quite over a dozen persons.
This just leads me to think that anyone could pretty much charter someone, anyone's boat from any port in Taiwan, to within a reasonably distanced destination for a not-too-expensive price.
Taiwan is known locally as "Ghost Island" to some.