How to restore packages on the build server?


I migrated away from MSBuild- based NuGet package restore, to use Automatic Package Restore, and everything sounds perfect on my local machine, but seems like impossible to get the nugetter process succeed on the build-server, without committing the packages folder to the source control!
Since I've got to delete the Packages folder from source control, and set the disableSourceControlIntegration settings to "true", shouldn't the nugetter template execute the command "nuget.exe restore" before starting the build?
Is there any work around for this? Any plans to add this feature to the nugetter template?