 Red Cross workers visit detainees in 80 countries |
The state department's top legal adviser, John Bellinger, made the admission but gave no details about where such prisoners were held.
Correspondents say the revelation is likely to increase suspicion that the CIA has been operating secret prisons outside international oversight.
The issue has dogged Secretary of State Condoleezza Rice's tour to Europe.
Mr Bellinger made the admission in Geneva.
He stated that the group International Committee of the Red Cross (ICRC) had access to "absolutely everybody" at the prison camp in Guantanamo Bay, Cuba, which holds suspects detained during the US war on terror.
When asked by journalists if the organisation had access to everybody held in similar circumstances elsewhere, he said: "No". He declined to explain further.
Until now the US administration has been careful in its language, says the BBC's state department correspondent Jonathan Beale.
It has always said that the ICRC has access to all prisoners held at US defence department facilities - leaving open the question of whether there are CIA prisons elsewhere.
Allegations 'ludicrous'
Mr Bellinger's comments will raise suspicions that high-profile terrorist suspects are being held out of international view, our correspondent says.
Mr Bellinger said some of the allegations of secret prisons were "so overblown as to be ludicrous".
The ICRC wants access to all foreign terror suspects held by the US "in undisclosed locations".
"The dialogue continues on the question. We would like to obtain information and access to them," ICRC spokesman Florian Westphal said on Thursday.
Human rights groups say there is no way of knowing whether detainees being held in secret are being tortured.
On her visit to Europe, Condoleezza Rice has repeatedly denied that the US tortures prisoners.
On Wednesday, Ms Rice stressed that all American interrogators were bound by the UN Convention on Torture, whether they worked in the US or abroad.
Nato and EU foreign ministers, after meeting Ms Rice in Brussels on Wednesday evening, declared themselves satisfied with her assurances that the US does not interpret international humanitarian law differently from its allies.
