A methodology is proposed for taking queries and requests expressed in natural language as input, and answering them in charts by organizing that interaction into appropriate dialogue. The objective of the research is to make it possible to interactively use charts and graphics, which, as well as languages, are important modes of communication, for analyzing huge amounts of data and for discovering its characteristics and tendencies. This paper shows that in such situations the correctness of the charts could be revealed by understanding the context, and proposes a framework to handle chart appropriateness through context processing. The core of the framework is a logical form that includes the specifications of users' perspectives and the proper treatment of the logical form for handling utterances and utterance fragments in contexts. The framework has been confirmed through an empirical study to be appropriate, and has been implemented.