The consumer confidence index fell in May to 102.3, down from a revised 103.7 in April, according to the Conference Board.
This was slightly above the median forecast in a MarketWatch survey of economists.
"Consumer confidence declined in May as consumers' view of current conditions became somewhat less upbeat while their expectations remained gloomy," the Conference Board's senior economics director Ataman Ozyildirim said in a statement.
"Their assessment of current employment conditions saw the most significant deterioration," Ozyildirim said, adding that the proportion of consumers reporting jobs are "plentiful" fell sharply in May.
"Consumers also became more downbeat about future business conditions," he added.
The most notable decline in consumer confidence came from consumers over 55 years of age, according to the Conference Board.
Despite consumers' concerns about the health of the job market, the U.S. economy added more than 250,000 jobs last month, and the unemployment rate fell to a low last seen in the 1960s.
The slight fall in consumer confidence will provide the Federal Reserve with a useful data point on how consumers view the economy as it mulls raising interest rates again in order to control rising prices.
Futures traders now assign a greater-than 60 percent chance the U.S. Federal Reserve (central bank) will vote to do raise its benchmark lending rate by a quarter percentage point at its next meeting on June 13-14, according to data from CME Group.