Hi Mark,
You're right, there is no way to load a driver as part of the installer. Instead you must inject the driver into the install disc as described in the following KB article.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2005205#Prior to a new ESXi installation
You may also find that your manufacturer (e.g. Dell, HP) provides customised install media.
Hope that helps.