In Sentinel Hub Configuration Utility for Sentinel S2: How do I make a layer that includes reflectance of all the bands? There does not seem to be such template in the list of EO Products.
There are two ways of achieving this.
As you have correctly identified, one is by configuring such layer in the configurator tool, second would be by using the power of custom scripts from the python package itself.
Both possibilities boil down to writing your own custom script, in this case quite simple one: return [B01, B02, B03, B04, B05, B06, B07, B08, B8A, B09, B10, B11, B12];