Observations of the Hubble Deep Field with the Infrared Space Observatory .4. Association of sources with Hubble Deep Field galaxies