In organic solar cells, the resulting device efficiency depends strongly on the local morphology and intermolecular interactions of the blend film. Optical spectroscopy was used to identify the spectral signatures of interacting chromophores in blend films of the donor polymer PM6 with two state-of-the-art nonfullerene acceptors, Y6 and N4, which differ merely in the branching point of the side chain. From temperature-dependent absorption and luminescence spectroscopy in solution, it is inferred that both acceptor materials form two types of aggregates that differ in their interaction energy. Y6 forms an aggregate with a predominant J-type character in solution, while for N4 molecules the interaction is predominantly in a H-like manner in solution and freshly spin-cast film, yet the molecules reorient with respect to each other with time or thermal annealing to adopt a more J-type interaction. The different aggregation behavior of the acceptor materials is also reflected in the blend films and accounts for the different solar cell efficiencies reported with the two blends.