You may not be planting the types of flowers that butterflies like and can access easily. Here are some plants the butterflies have enjoyed in my garden:
Oregano: butterflies will absolutely cover the plant when in flower
Mints of all kinds
Butterfly Bush (buddleia)
Sweet Rocket (Hesperis Matronalis)
Lilacs
Coneflower (Echinacea)
Butterfly Weed (Asclepias)
Black Eyed Susan (Rudbeckia)
...among many, many others
There are web sites and books devoted to butterfly gardening.
Other considerations include, are you planting and setting aside for butterfly young (caterpillars) to develop and feed?
Are you providing a water source for butterflies to drink?
Does your site provide plenty of sun?