Two of the U.S. swimmers who were with Ryan Lochte during his alleged robbery in Rio were removed from their plane by Brazilian authorities on Wednesday night, the U.S. Olympic Committee confirmed.

Authorities are investigating whether Jack Conger and Gunnar Bentz, who were both escorted off the plane, were actually robbed at gunpoint early Sunday in Rio de Janeiro as they claimed to the press, along with Lochte and swimmer James Feigen.

A judge ordered Wednesday for Lochte to stay in Brazil and issued a search and seizure warrant for the swimmer. Lochte, however, had already left the country before police could seize his passport and ensure that he could answer questions in the investigation. It’s unclear if Feigen, for which a warrant was also issued, is still in the country.

Lochte flew out of Brazil on Monday, before the order was issued, federal police said. Authorities apparently showed up at the Olympic village on Wednesday, only to find that the swimmers were not there for questioning.

“The swim team moved out of the village after their competition ended, so we were not able to make the athletes available,” said U.S. Olympic Committee spokesman Patrick Sandusky said at the time.

“We will continue to cooperate with the Brazilian authorities,” he added.

Lochte and the American swimmers claimed to have been robbed at gunpoint early Sunday after leaving a party, when the taxi they were in was pulled over by criminals posing as police, the U.S. Olympic Committee said in a statement. In an interview with “Today’s” Billy Bush, Lochte described having a gun pointed at his forehead as the robbers stole their wallets.

Lochte’s mother also confirmed the story to USA Today.

It was a story that raised even more concerns over the safety of athletes in Rio de Janeiro. However, in recent days, the incident has been placed under a microscope, with some questioning the veracity of the report.

Rio authorities said they have not found the driver of the taxi the men were allegedly in, and have had trouble finding witnesses to corroborate their story.

A video reportedly showing the athletes returning to the Olympic village shows what appears to be their wallets and phones, which they said were taken during the robbery, in their possession walking through metal detectors.

“You can see the supposed victims arriving without signs of being physically or psychologically shaken, even joking amongst themselves,” Judge Keyla Blanc de Cnop said in a statement to the New York Times.

Lochte, a 12-time Olympic gold medalist, won a gold medal in the 4×200 freestyle relay.

Pictured above: U.S. Olympic swimmers Gunnar Bentz and James Feigen stand together during the men’s 400-meter relay team medal ceremony in Rio.

2016 Olympics: The Biggest Controversies to Come Out of Rio