The two main modern traditions of Easter -- Easter eggs and the Easter bunny -- are both ancient symbols of rebirth and new life, associated with the return of the season of growth after the long winter. The connection of eggs with new life is clear. Zoroastrians in Persia, well over 2,000 years ago, painted eggs as part of the celebration of Nôrûz (the Persian new year, which falls on the spring equinox), so this practice is at least that old. Rabbits and hares, too, have long been fertility symbols due to their rapid breeding.
In the lands of northern Europe, Easter was especially significant in pre-Christian times because of the harshness of the winters (the onset of winter, recognized at Samhain or "Halloween", was equally important). Oddly enough, the origin of the name "Easter" is rather mysterious. It's based on the name of a pagan Germanic goddess of spring, "Eostre", whose sign was a hare and who is described in the writings of a medieval chronicler known as the Venerable Bede. German linguist Jakob Grimm postulated a related continental Germanic goddess whose name he reconstructed as "Ostara". But neither Eostre nor Ostara is attested in other sources, and it's not clear whether this goddess was, in fact, really part of old Germanic paganism at all, or just an invention of Bede.
It has also been claimed that the name "Easter" is derived from the pagan Babylonian goddess Ishtar; this would have been fitting since Ishtar's consort Tammuz (Sumerian Damu-zid) was said to have died but later been allowed to return to the land of the living, a tale which served as a template for many later Middle Eastern myths of dying and resurrected deities, including the Christian one. But in fact there's no evidence of any connection, and the similarity of names seems to be merely coincidence. Springtime celebrations of rebirth are a common feature of human cultures and must have developed independently in many places.
Like so many pagan observances, the northern European Easter was eventually co-opted by the usurping alien religion from the Middle East. But the egg hunts and the Easter bunny have never been successfully imbued with any Christian significance, while their true pagan origins lie too far in the past to have any modern meaning. With candy eggs now used in place of real ones, not even chickens need be offended. Happy hunting!