The number of households using stand-alone conversational agents is rapidly increasing. However, recent research revealed that in some of these households, use of agents decreases over time, and we know little about why. Therefore, we aim to understand how people utilize such devices in their daily lives and to explore the associated obstacles or difficulties. We conducted a long-term (12-week) user study in which we followed eight households, examining their use of Amazon Echo at home. From a series of diaries, surveys, and interviews with eight first-time users, we identified how their experiences changed over time and how conversational agents lose their presence at home. We found that voice interface, physical form, and at-home installation affect users' perceptions and expectations of smart speakers. Based on these findings, we discuss challenges and design opportunities for future at-home conversational agents.