How to load asset gems only for precompile on heroku?

Rails 4.1+, so there isn't built-in support for an :assets group

I want to keep Heroku's precompile on push behaviour, but don't want the asset gems loaded by the rails server. We don't use any kind of inline coffeescript or scss template rendering in the app, only in the assets, so it's just wasted memory at runtime.

I've played around with extending the rake task, configuring sprocket-rails, and even changing application.js to application.js.erb and adding things like

//= <% require 'jquery-rails' %> 
//= require 'jquery'

but still get these errors:

Sprockets::FileNotFound: couldn't find file 'jquery'

If I keep the asset gems in the default Gemfile group everything works fine.

The point here is to not have them loaded in the production environment, but to have

RAILS_ENV=production rake assets:precompile task 

load them before it executes (and fails because of missing libraries)