Ten fire appliances and about 50 firefighters are at the scene on Foley Street, in the Attercliffe area of the city after being called at 14:30 BST.
A spokesperson for South Yorkshire Fire and Rescue Service said early reports were that rubbish outside had caught fire and the flames spread to the Sheffield Reclaim Ltd building.
There are no reports of casualties but a large plume of smoke can be seen.
The fire service said about 20 tonnes of plastic was involved.
Incident commander Rick Rolt said: \”Although there was a lot of black smoke, the plume mostly blew over an industrial area and we\’re confident it wasn\’t hazardous.\”
He added that though they hoped to scale back the number of crews at the scene, they could be there for a number of days.
\”We\’re likely to be turning the plastics over and damping down for days but we\’ve just got monitor jets on the fire now so we may be able to scale back later.
\”The crews have done an absolutely fantastic job.
\”We were really worried about the fire spreading to neighbouring buildings such as Veolia. Then there would have been lots of potentially hazardous problems and we\’d have been looking at evacuations.
\”But the crews put some fire breaks in to stop the fire spreading, and they managed to contain it to the premise involved,\” he said.
The fire service has advised people living and working near the area to go indoors and keep their windows closed.
South Yorkshire Police said road closures were in place around the area as firefighters tackled the blaze.