Recent advancements in Artificial Intelligence, particularly in Large Language Models (LLMs) and Vision-Language Models (VLMs), have opened new possibilities for improving human-robot interaction. However, enabling robots to understand natural language instructions and accurately execute them in dynamic environments remains a major challenge. Issues like hallucinated output from language models and inaccurate object localization limit the use of AI in real-world robotic applications. This thesis addresses these challenges by proposing a system that generates action plans and robot control code for pick-and-place tasks based on natural language input.

The system integrates LLMs and VLMs into a three-layer framework: an Interpreter layer to analyze user queries, an Object Localizer layer to find the referred object in a simulated environment, and a Responder layer to generate an executable code. The system was tested using a UR5 robot in simulation, and two VLMs, CLIPSeg and Grounding DINO, were compared and modified for improved accuracy. Additionally, Retrieval- Augmented Generation (RAG) was implemented to provide external documents to the LLM, reducing hallucination, and improving code quality. The results show that the system can reliably interpret user queries and perform pick-and-place tasks, forming a solid foundation for expanding to more complex robotic behaviors.