ES详解 - 资料：Awesome Elasticsearch

本文来自 GitHub Awesome Elasticsearch 项目 (opens new window), 搜集ElasticSearch相关的优秀资料。@pdai

General

Elastic Stack

• Elasticsearch (opens new window) official website
• Logstash (opens new window) is a data pipeline that helps you process logs and other event data from a variety of systems
• Kibana (opens new window) is a data analysis tool that helps to visualize your data; Kibana Manual docs (opens new window)
• beats (opens new window) is the platform for building lightweight, open source data shippers for many types of data you want to enrich with Logstash, search and analyze in Elasticsearch, and visualize in Kibana.

Books

• Deep Learning for Search (opens new window) - teaches you how to leverage neural networks, NLP, and deep learning techniques to improve search performance. (2019)
• Relevant Search: with applications for Solr and Elasticsearch (opens new window) - demystifies relevance work. Using Elasticsearch, it teaches you how to return engaging search results to your users, helping you understand and leverage the internals of Lucene-based search engines. (2016)
• Elasticsearch in Action (opens new window) - teaches you how to build scalable search applications using Elasticsearch (2015)

Related (awesome) lists

• frutik/awesome-search (opens new window) I am building e-commerce search now. Below are listed some of my build blocks

Open-source and free products, based on Elasticsearch

• Fess (opens new window) is an open source full featured Enterprise Search, with a web-crawler
• Yelp/elastalert (opens new window) is a modular flexible rules based alerting system written in Python
• etsy/411 (opens new window) - an Alert Management Web Application https://demo.fouroneone.io (credentials: user/user)
• appbaseio/mirage (opens new window) is a ???? GUI for composing Elasticsearch queries
• exceptionless/Exceptionless (opens new window) is an error (exceptions) collecting and reporting server with client bindings for a various programming languages
• searchkit/searchkit (opens new window) is a UI framework based on React to build awesome search experiences with Elasticsearch
• appbaseio/reactivemaps (opens new window) is a React based UI components library for building Airbnb / Foursquare like Maps
• appbaseio/reactivesearch (opens new window) is a library of beautiful React UI components for Elasticsearch
• appbaseio/dejavu (opens new window) The missing UI for Elasticsearch; landing page (opens new window)
• Simple File Server (opens new window) is an Openstack Swift compatible distributed object store that can serve and securely store billions of large and small files using minimal resources.
• logagent (opens new window) a log shipper to parse and ship logs to Elasticsearch including bulk indexing, disk buffers and log format detection.
• ItemsAPI (opens new window) simplified search API for web and mobile (based on Elasticsearch and Express.js)
• Kuzzle (opens new window) - An open-source backend with advanced real-time features for Web, Mobile and IoT that uses ElasticSearch as a database. (Website (opens new window))
• SIAC (opens new window) - SIAC is an enterprise SIEM built on the ELK stack and other open-source components.
• Sentinl (opens new window) - Sentinl is a Kibana alerting and reporting app.
• Praeco (opens new window) - Elasticsearch alerting made simple

Elasticsearch developer tools and utilities

Development and debugging

• Sense (from Elastic) (opens new window) A JSON aware developer console to Elasticsearch; official and very powerful
• ES-mode (opens new window) An Emacs major mode for interacting with Elasticsearch (similar to Sense)
• Elasticsearch Cheatsheet (opens new window) Examples for the most used queries, API and settings for all major version of Elasticsearch
• Elasticstat (opens new window) CLI tool displaying monitoring informations like htop
• Elastic for Visual Studio Code (opens new window) An extension for developing Elasticsearch queries like Kibana and Sense extention in Visual Studio Code
• Elastic Builder (opens new window) A Node.js implementation of the Elasticsearch DSL
• Bodybuilder (opens new window) A Node.js elasticsearch query body builder
• enju (opens new window) A Node.js elasticsearch ORM
• Peek (opens new window) An interactive CLI in Python that works like Kibana Console with additional features

Import and Export

• Knapsack plugin (opens new window) is an "swiss knife" export/import plugin for Elasticsearch
• Elasticsearch-Exporter (opens new window) is a command line script to import/export data from Elasticsearch to various other storage systems
• esbulk (opens new window) Parallel elasticsearch bulk indexing utility for the command line.
• elasticdump (opens new window) - tools for moving and saving indices
• elasticsearch-loader (opens new window) - Tool for loading common file types to elasticsearch including csv, json, and parquet

Management

• Esctl (opens new window) - High-level command line interface to manage Elasticsearch clusters.
• Vulcanizer (opens new window) - Github's open sourced cluster management library based on Elasticsearch's REST API. Comes with a high level CLI tool

Elasticsearch plugins

Cluster

• sscarduzio/elasticsearch-readonlyrest-plugin (opens new window) Safely expose Elasticsearch REST API directly to the public
• mobz/elasticsearch-head (opens new window) is a powerful and essential plugin for managing your cluster, indices and mapping
• Bigdesk (opens new window) - Live charts and statistics for elasticsearch cluster
• Elastic HQ (opens new window) - Elasticsearch cluster management console with live monitoring and beautiful UI
• Cerebro (opens new window) is an open source(MIT License) elasticsearch web admin tool. Supports ES 5.x
• Kopf (opens new window) - Another management plugin that have REST console and manual shard allocation
• Search Guard (opens new window) - Elasticsearch and elastic stack security and alerting for free
• ee-outliers (opens new window) - ee-outliers is a framework to detect outliers in events stored in an Elasticsearch cluster.
• Elasticsearch Comrade (opens new window) - Elasticsearch admin panel built for ops and monitoring
• elasticsearch-admin (opens new window) - Web administration for Elasticsearch

Other

• SIREn Join Plugin for Elasticsearch (opens new window) This plugin extends Elasticsearch with new search actions and a filter query parser that enables to perform a "Filter Join" between two set of documents (in the same index or in different indexes).

Integrations and SQL support

• NLPchina/elasticsearch-sql (opens new window) - Query elasticsearch using familiar SQL syntax. You can also use ES functions in SQL.
• elastic/elasticsearch-hadoop (opens new window) - Elasticsearch real-time search and analytics natively integrated with Hadoop (and Hive)
• jprante/elasticsearch-jdbc (opens new window) - JDBC importer for Elasticsearch
• pandasticsearch (opens new window) - An Elasticsearch client exposing DataFrame API
• monstache (opens new window) - Go daemon that syncs MongoDB to Elasticsearch in near realtime

You know, for search

• jprante/elasticsearch-plugin-bundle (opens new window) A plugin that consists of a compilation of useful Elasticsearch plugins related to indexing and searching documents

Kibana plugins and applications

• elastic/timelion (opens new window) time-series analyses application. Overview and installation guide: Timelion: The time series composer for Kibana (opens new window)
• Kibana Alert App for Elasticsearch (opens new window) - Kibana plugin with monitoring, alerting and reporting capabilities
• VulnWhisperer (opens new window) - VulnWhisperer is a vulnerability data and report aggregator.
• Wazuh Kibana App (opens new window) - A Kibana app for working with data generated by Wazuh (opens new window).
• Datasweet Formula (opens new window) - A real time calculated metric plugin Datasweet Formula (opens new window).

Kibana Visualization plugins

• nbs-system/mapster (opens new window) - a visualization which allows to create live event 3d maps in Kibana
• Kibana Tag Cloud Plugin (opens new window) - tag cloud visualization plugin based on d3-cloud plugin
• LogTrail (opens new window) - a plugin for Kibana to view, analyze, search and tail log events from multiple hosts in realtime with devops friendly interface inspired by Papertrail
• Analyze API (opens new window) - Kibana 6 application to manipulate the _analyze API graphically
• kbn_network (opens new window) - This is a plugin developed for Kibana that displays a network node that link two fields that have been previously selected.

Discussions and social media

• /r/elasticsearch (opens new window)
• Elasticsearch forum (opens new window)
• Stackoverflow (opens new window)
• Books on Amazon (opens new window) does not fit well into this category, but worth checking out!
• TODO: Put some good twitter accounts

Tutorials

• Centralized Logging with Logstash and Kibana On Ubuntu 14.04 (opens new window) everything you need to now when you are creating your first Elasticsearch+Logstash+Kibana instance
• dwyl/learn-elasticsearch (opens new window) a getting started tutorial with a pack of valuable references
• Make Sense of your Logs: From Zero to Hero in less than an Hour! by Britta Weber (opens new window) demonstrates how you can build Elasticsearch + Logstash + Kibana stack to collect and discover your data
• $$ Elasticsearch 7 and Elastic Stack (opens new window) - liveVideo course that teaches you to search, analyze, and visualize big data on a cluster with Elasticsearch, Logstash, Beats, Kibana, and more.

Articles

System configuration

• A Useful Elasticsearch Cheat Sheet in Times of Trouble (opens new window)
• The definitive guide for Elasticsearch on Windows Azure (opens new window)
• Elasticsearch pre-flight checklist (opens new window)
• 9 Tips on Elasticsearch Configuration for High Performance (opens new window)
• Best Practices in AWS (opens new window)
• How to Secure Elasticsearch and Kibana (opens new window) with NGINX, LDAP and SSL ????
• Elasticsearch server on Webfaction using NGINX with basic authorization and HTTPS protocol (opens new window)
• Elasticsearch Guides (opens new window) Useful Elasticsearch guides with best practices, troubleshooting instructions for errors, tips, examples of code snippets and more.

Docker and Elasticsearch

• Running an Elasticsearch cluster with Docker (opens new window)

Java tuning

• Elasticsearch Java Virtual Machine settings explained (opens new window)
• Tuning Garbage Collection for Mission-Critical Java Applications (opens new window)
• G1: One Garbage Collector To Rule Them All (opens new window)
• Use Lucene’s MMapDirectory on 64bit platforms, please! (opens new window)
• Black Magic cookbook (opens new window)
• G1GC Fundamentals: Lessons from Taming Garbage Collection (opens new window)
• JVM Garbage Collector settings
  investigation (opens new window) PDF Comparison of JVM GC
• Garbage Collection Settings for Elasticsearch Master Nodes (opens new window) Fine tunine your garbage collector
• Understanding G1 GC Log Format (opens new window) To tune and troubleshoot G1 GC enabled JVMs, one must have a proper understanding of G1 GC log format. This article walks through key things that one should know about the G1 GC log format.

How to start using G1

#ES_JAVA_OPTS=""
ES_JAVA_OPTS="-XX:-UseParNewGC -XX:-UseConcMarkSweepGC -XX:+UseG1GC"

Scalable Infrastructure and performance

• The Authoritative Guide to Elasticsearch Performance Tuning (Part 1) (opens new window) Part 2 (opens new window) Part 3 (opens new window)
• Tuning data ingestion performance for Elasticsearch on Azure (opens new window) - and not only for Azure. That's a great article about Elasticsearch Performance testing by example
• Elasticsearch Indexing Performance Cheatsheet (opens new window) - when you plan to index large amounts of data in Elasticsearch (by Patrick Peschlow)
• Elasticsearch for Logging (opens new window) Elasticsearch configuration tips and tricks from Sanity
• Scaling Elasticsearch to Hundreds of Developers (opens new window) by Joseph Lynch @yelp
• 10 Elasticsearch metrics to watch (opens new window)
• Understanding Elasticsearch Performance (opens new window)
• Our Experience of Creating Large Scale Log Search System Using Elasticsearch (opens new window) - topology, separate master, data and search balancers nodes
• ???? Elasticsearch on Azure Guidance (opens new window) it is 10% on Azure and 90% of a very valuable general information, tips and tricks about Elasticsearch
• How to avoid the split-brain problem in Elasticsearch (opens new window)
• Datadog's series about monitoring Elasticsearch performance:
  • How to monitor Elasticsearch performance (opens new window)
  • How to collect Elasticsearch metrics (opens new window)
  • How to monitor Elasticsearch with Datadog (opens new window)
  • How to solve 5 Elasticsearch performance and scaling problems (opens new window)
• Performance Monitoring Essentials - Elasticsearch Edition (opens new window)
• Operator for running Elasticsearch in Kubernetes (opens new window)

Integrations

• Apache Hive integration (opens new window)
• Connecting Tableau to Elasticsearch (READ: How to query Elasticsearch with Hive SQL and Hadoop) (opens new window)
• mradamlacey/elasticsearch-tableau-connector (opens new window)

Logging

• 5 Logstash Alternatives (opens new window) and typical use cases

Alerts

• ElastAlert: Alerting At Scale With Elasticsearch, Part 1 (opens new window) by engineeringblog.yelp.com
• ElastAlert: Alerting At Scale With Elasticsearch, Part 2 (opens new window) by engineeringblog.yelp.com
• Elastalert: implementing rich monitoring with Elasticsearch (opens new window)

Time series

• Elasticsearch as a Time Series Data Store (opens new window) by Felix Barnsteiner
• Running derivatives on Voyager velocity data (opens new window) By Colin Goodheart-Smithe
• Shewhart Control Charts via Moving Averages: Part 1 (opens new window) - Part 2 (opens new window) by Zachary Tong
• Implementing a Statistical Anomaly Detector: Part 1 (opens new window) - Part 2 (opens new window) - Part 3 (opens new window) by Zachary Tong

Machine Learning

• Classifying images into Elasticsearch with DeepDetect (opens new window) (forum thread with discussion (opens new window)) by Emmanuel Benazera
• Elasticsearch with Machine Learning (opens new window) (English translation (opens new window)) by Kunihiko Kido
• Recommender System with Mahout and Elasticsearch (opens new window)

Use cases for Elasticsearch

• Data Infrastructure at IFTTT (opens new window) Elasticsearch, Kafka, Apache Spark, Redhsift, other AWS services
• OFAC compliance with Elasticsearch (opens new window) using AWS
• Building a Streaming Search Platform (opens new window) -
  Streaming Search on Tweets: Storm, Elasticsearch, and Redis

Other

• LogZoom, a fast and lightweight substitute for Logstash (opens new window)
• Graylog2/graylog2-server (opens new window) - Free and open source log management (based on ES)
• Fluentd vs. Logstash for OpenStack Log Management (opens new window)
• Building a Directory Map With ELK (opens new window)
• Structured logging with ELK - part 1 (opens new window)
• Search for ???? Emoji with Elasticsearch ???? (opens new window)
• Complete Guide to the ELK Stack (opens new window)
• logiq (opens new window) - Simple WebUI Monitoring Tool for Logstash ver. 5.0 and up
• ElasticSearch Report Engine (opens new window) - An ElasticSearch plugin to return query results as either PDF,HTML or CSV.
• Elasticsearch Glossary (opens new window) - explanations of Elasticsearch terminology, including examples, common best practices and troubleshooting guides for various issues.

Videos

Overviews

• Elasticsearch for logs and metrics: A deep dive – Velocity 2016 (opens new window) by Sematext Developers
• Elasticsearch in action Thijs Feryn (opens new window) a beginner overview
• Getting Down and Dirty with ElasticSearch by Clinton Gormley (opens new window)
• How we scaled Raygun (opens new window)
• Getting started with Elasticsearch (opens new window)
• Speed is a Key: Elasticsearch under the Hood (opens new window) introduction + basic performance optimization
• $$ Pluralsight: Getting Started With Elasticsearch for .NET Developers (opens new window) this course will introduce users to Elasticsearch, how it works, and how to use it with .NET projects.
• $$ Complete Guide to Elasticsearch (opens new window) Comprehensive guide to Elasticsearch, the popular search engine built on Apache Lucene
• How Elasticsearch powers the Guardian's newsroom (opens new window)
• Elasticsearch Query Editor in Grafana (opens new window)
• Scale Your Metrics with Elasticsearch (opens new window) 2019 by Philipp Krenn (Elastic) optimization tips and tricks

Advanced

• #bbuzz 2015: Adrien Grand – Algorithms and data-structures that power Lucene and Elasticsearch (opens new window)
• Rafał Kuć - Running High Performance Fault-tolerant Elasticsearch Clusters on Docker (opens new window) and slides (opens new window)
• Working with Elasticsearch - Search, Aggregate, Analyze, and Scale Large Volume Datastores (opens new window) - O'Reilly Media
• End-to-end Recommender System with Spark and Elasticsearch (opens new window) by Nick Pentreath & Jean-François Puget. Slide deck (opens new window)

Code, configuration file samples and other gists

• Elasticsearch config for a write-heavy cluster (opens new window) - reyjrar/elasticsearch.yml
• chenryn/ESPL - Elastic Search Processing Language (opens new window) PEG parser sample for SPL to Elasticsearch DSL
• thomaspatzke/EQUEL (opens new window) an Elasticsearch QUEry Language, based on G4 grammar parser

Who is using elasticsearch?

Yelp (opens new window),
IFTTT (opens new window),
StackExchange (opens new window),
Raygun (opens new window),
Mozilla (opens new window),
Spotify (opens new window),
CERN (opens new window),
NASA (opens new window) Zalando (opens new window)

I want more! (Elasticsearch related resources)

• Technology Explained Blog (opens new window)
• EagerElk (opens new window)
• Tim Roes Blog (opens new window)

Contributing

• Make sure you are about to post a valuable resource that belongs to this list
• Do NOT group ++Add and --Remove changes in same PR. Make them separate pull requests
• Use spellchecker
• All spelling and grammar corrections are welcome (except for the rule above)
• Fork this repo, do your edits, send the pull request
• Feel free to create any new sections
• Do not even try to add this repo to any awesome-awesome-* lists

← Awesome TypeScript (opens new window) -= Awesome Elasticsearch =-