Do I need to have poor, infertile soil to successfully create a wild flower area?

No, this is something of a misconception. Wild flowers will grow well on fertile soils. However it is the rapid growth response of other competitive species, particularly grasses that can cause problems on very fertile sites. If you prepare the site adequately before seeding you should have little problem. It is usually agricultural land which has had a lot of manure or fertilisers applied in the past that can be a challenge. It is the Phosphorus level within the soil which is the main element I test for as an indication of fertility. Generally speaking I would not attempt to convert an area with a Phosphorus Index of 3 or above (26-45 ppm in the soil), unless it was for an annual wild flower meadow (as my annual wild flower mix would do well in this situation). I have successfully created and managed excellent perennial wild flower meadows with Phosphorus levels of 2.7 and below. Other elements must also be taken into consideration in addition to the fertility of a site, such as the amount of direct sunlight it receives, how well draining the soil is etc. These all come into play as factors either favouring the wild flowers or their competition.