Some aspects of building will be impossible during the winter. For example, you will not be able to dig out the foundation if the ground is frozen solid. Another important consideration is protecting the foundation against the cold. This is something that you will need to discuss with your builder.
An advantage to winter building is that it is the slow season and work crews will be available. Also, your contractor will probably be able to focus all his attention and resources on you.
If you are planning to sell your house, spring is the best time to sell. Completing the construction during the winter will have the house available when the market goes up in the spring. This might be something you consider.

Spring Building
We all know that April showers bring May flowers and those April showers are main concern for people building during the spring. You will need to protect your home from water damage and even flood damage during the rainy season. This is usually accomplished by getting the shingles on the roof as soon as possible during the springtime.
The other thing that the rain brings is mud. Just forget about a clean construction site if you are going to build during the spring. You will need to consider which parts of the building get finished during the spring (drywall yes, plumbing yes) and which aspects are not such a great idea (landscaping no, carpeting no). Common sense and an experienced builder will help you make good decisions.

Summer building
This is the height of the construction season. The weather is nice and you will likely lose very few workdays if you are building during the summer. Also, you won't need to worry so much about your materials blowing away in a storm or being damaged by rain.
The only disadvantage to building in the summer is that it is when everybody else is building. This might tax the time and resources of your builder. When choosing a builder find out how many other projects they will be handling during your construction process.

Fall Building
In most places, the fall is an excellent time to build. There is a hurricane season to contend with along the gulf coast and that might be a consideration. Typically however, fall weather is beautiful and it is a good time to build. One thing to consider when building in the fall is that delays will push you toward the winter season. If you are concerned about building in the winter, you should be finishing your project in the fall rather than beginning it in the fall.
If you do begin your project in the fall, it is critical that your foundation is laid before the ground freezes. This is another aspect that you should discuss with your builder.