Building a Conversational Interface for Elasticsearch Data with Kestra and OpenAI

To create natural language interactions with Elasticsearch data, use Kestra with the OpenAI plugin. This combination transforms structured data into an intuitive, conversational interface.

Key advantages:

• Custom data processing and AI prompts for your specific use case
• Easy configuration and deployment with Kestra’s YAML-based workflow

Here’s a code example demonstrating this workflow:

id: movie_recommendation_system
namespace: entertainment.movies

inputs:
  - id: user_preference
    type: STRING
    defaults: I like action movies with a bit of comedy

tasks:
  - id: search_movies
    type: io.kestra.plugin.elasticsearch.Search
    connection:
      hosts: 
        - http://localhost:9200/
    indexes:
      - movies_database
    request:
      size: 5
      query: 
        bool:
          must:
            multi_match:
              query: "{{ inputs.user_preference }}"
              fields: ["title", "description", "genre"]
              type: best_fields

  - id: format_movie_results
    type: io.kestra.plugin.core.debug.Return
    format: >
      {% for movie in outputs.search_movies.rows %}
        Title: {{ movie.title }}
        Genre: {{ movie.genre }}
        Description: {{ movie.description }}
      {% endfor %}

  - id: generate_recommendations
    type: io.kestra.plugin.openai.ChatCompletion
    apiKey: sk-proj-your-OpenAI-API-KEY
    model: gpt-4
    maxTokens: 500
    prompt: |
      You're a movie recommendation assistant. 
      Based on the USER PREFERENCE and the MOVIE RESULTS, suggest 3 movies from the list.
      Explain why each movie matches the user's preference.
      USER PREFERENCE: {{ inputs.user_preference }}
      MOVIE RESULTS: {{ outputs.format_movie_results.value }}

  - id: display_recommendations
    type: io.kestra.plugin.core.log.Log
    message: "{{ outputs.generate_recommendations.choices | jq('.[].message.content') | first }}"

This example shows:

1. search_movies task uses Elasticsearch Search to query the index based on user preference.
2. format_movie_results task formats the search results.
3. generate_recommendations task uses OpenAI ChatCompletion to analyze results and user preference, creating personalized movie recommendations with explanations.