Since political contacts and clout were the lifeblood of the law firm, Sweeny entered local politics in the 1850s as a leader in the Twentieth Ward and a member of the general committee of the Tammany Society, the main Democratic organization in New York City. An intelligent but shy man, Sweeny preferred back-room negotiations to stumping for office. In 1854 he served as a lobbyist at the state capital for various utility and transportation companies and banks. He also served on several of their boards and became rich from investing in them. He did run for office once, winning the position of district attorney in 1857. But he suffered an emotional breakdown during his first jury trial and consequently resigned from office.

Ironically, like urban reformers, Sweeny despised what he perceived as the infighting and inefficiency of democratic politics and sought to impose centralized control to expand and improve city services like in Paris and other European cities. But his method of accomplishing that goal was to help create a dominant political machine-Tammany Hall-based on patronage and favor-granting to businessmen, investors, judges, and politicians, practices that reformers characterized as corrupt. Sweeny became known as the "Brains" behind "Boss" William Tweed.

By 1863 Sweeny dominated Tammany's general committee, and in 1866 was appointed city chamberlain (i.e., treasurer) by Mayor John Hoffman. In the past, the unsalaried chamberlains had appropriated the interest off invested city monies for their compensation. Sweeny, however, abandoned that ethically and legally dubious custom in lieu of a nominal salary. He managed the mayoral nomination and victory of A. Oakey Hall in 1868. A new city charter concentrated power in a board consisting of Hall, Sweeny, and Tweed. Greatly expanding contracted city services opened the possibility of taking kickbacks and making investments in the franchised monopolies. When he was named director of the parks department in 1870, Sweeny switched funding from Central Park to parks in working-class neighborhoods and swelled the employment roll.