ELK / ESS Engineer

Key Feature Required

• Query tuning, indexing strategy, cluster monitoring, and troubleshooting, often using the ELK stack.
• Designing, implementing, and managing search and analytics solutions using Elasticsearch.
• Responsibilities may include indexing large datasets, optimizing search queries, maintaining cluster performance, and ensuring data availability.
• Single person in requirement backlog and direct interaction with client.

Key Responsibilities

• Visualization Creation: Build and assemble interactive panels, charts, maps, and metrics using Kibana Lens to create comprehensive dashboards.
• Data Analysis & Mapping: Design efficient time-series index mappings and data streams to ensure optimal data storage and retrieval.
• Query Optimization: Utilize aggregations, date histograms, and filters (KQL) to analyze large datasets and ensure fast dashboard response times.
• Alerting & Monitoring: Set up threshold-based alerts (Watcher) and monitor system health to provide actionable insights.
• Dashboard Optimization: Tune dashboard panels for performance, implementing data retention policies (ILM) to maintain efficiency.
• Cluster Management: Deploy, configure, and maintain Elasticsearch clusters on-premise or in cloud environments (AWS, Azure). [NICE TO HAVE]
• Performance Optimization: Fine-tune query performance, index management, and shard allocation for large-scale data. [NICE TO HAVE]
• Data Integration: Develop pipelines for indexing data from various sources using Logstash or ingestion APIs.
• Monitoring & Security: Monitor cluster health, maintain security protocols, and ensure data integrity.
• Troubleshooting: Perform root cause analysis on performance bottlenecks and cluster failures
• Work with Development Teams: Collaborate with software engineers to implement search features and improve user experiences.
• Provide Technical Support: Troubleshoot and resolve issues related to Elasticsearch performance, data integrity, and availability.
• Knowledge of indexing strategies for high-volume data.
• Experience in designing scalable, secure, and resilient search architectures.
• Ability to work INDEPENDENTLY in agile teams, collaborating with DevOps and Data Engineers
• Connect AI assistants, agents & automations to your data w/the first managed MCP platform [NICE TO HAVE]

Technical Skills

• Expertise in ELK Stack: Proficient in Elasticsearch, Logstash, and Kibana.
• Visualization Experience: Strong experience with Kibana visualization tools (Lens, Maps, Graph).
• Data Modelling and Knowledge of JSON and REST APIs: Familiarity with JSON data format and RESTful API principles is crucial for interacting with Elasticsearch.
• Querying: Proficiency in Kibana Query Language (KQL) and Elasticsearch aggregations.
• Monitoring/Observability: Background in creating operational dashboards for log analysis or metric tracking.
• Familiarity with Elasticsearch Ecosystem: Knowledge of related tools like Kibana, Logstash, and Beats enhances the engineer’s ability to deliver complete solutions.
• Basic Programming Skills: Proficiency in programming languages such as Python, Java, or Go is beneficial for automation and customization tasks.