The German government has so far refused to point the finger at Russia, despite the fact that the 49-year-old hitman holds a Russian passport.
"Russian authorities, despite repeated, high-level and insistent demands, did not participate enough in the investigation", the Foreign Ministry said in a statement.
"The foreign ministry has today declared two employees of the Russian embassy in Berlin as personae non grata with immediate effect", the ministry said in a statement.
Federal prosecutors in charge of intelligence cases said they had taken over the investigation from Berlin authorities.
Russia, which denies any involvement, responded swiftly, saying it would retaliate for what it called Germany's "unfriendly" move.
"There is sufficient factual evidence to suggest that the killing. was carried out either on behalf of state agencies of the Russian Federation or those of the Autonomous Chechen Republic", the prosecutors said.
In August this year, Chechen exile Zelimkhan Khangoshvili was shot dead in a central Berlin park in broad daylight by a man on a bicycle, who was identified by eyewitnesses and picked up by the police shortly afterwards.
The killing drew suspicion of Russian involvement from the beginning, although Moscow had denied any involvement. He was also known by a second identity "Tornike K." which was the one used by German prosecutors.
He applied for asylum in Germany in 2016 following multiple attempts on his life in Georgia. Prosecutors said he also went by the alias Vadim S., and German and global news outlets have reported he had links to organized crime in Russian Federation. Khangoshvili's ex-wife, Manana Tsatieva, previously told DW: "We were warned that this would happen eventually".
Russia, meanwhile, has pledged "retaliatory measures" following the expulsions.
Berlin's decision marks an escalation in already heightened tensions between Russia and Germany and other Western countries following the poisoning past year of a former Russian spy, Sergei Skripal, and his daughter on British soil.