Rocks rolled down a hillside in Thatichetlapalem, burying several houses. While three people died on the spot, another man died at the King George Hospital, the official said.
Four others were injured in the rockslide, the official added.
Heavy rains overnight were believed to have caused the disaster.
State minister for human resource G Srinivas Rao visited the site. Many houses have been built in areas vulnerable to landslides, he said.