Anything on the surface of the solar PV panels decreases the amount of sunlight (photons) striking the solar cells and will result in decreased energy production. Foreign particulate matter such as dust, weeds, pollen, bird droppings, tree leaves & sap, spider webs, moss or fungus, ash from fires and other contaminates. All of these decrease energy output.
If you live in an area with strong winds, your chances of accumulating dirt will be much higher. Dirt can also be transported through rain as it falls through the atmosphere picking up impurities in the air.