@beenthere
OK, diploids do apparently, on average, have strong negative effects of inbreeding. In crosses that you consider are significant inbreeding you can alleviate the potential problems by producing many more seeds than usual and by culling more heavily for characteristics that are "closer" to "fitness". Those would be pod fertility [percentage of flowers that produce pods with seeds (when pollinated with fertile pollen)], average number of seeds in those pods); pollen fertility (percentage of flowers that set pods (of known to be fertile pod parents); rate of fan growth; rate of increase; winter hardiness; robustness (for example percentage of small fans that survive being transplanted or that survive their first winter, etc.).
It may require you to be more ruthless about the number of seedlings that end in the compost pile, but you should be able to select for fitter individuals even in crosses that on average produce less fit seedlings - as long as enough seedlings are grown.
