ELK Stack with BizTalk: Part 2: Notes on installing ELK stack in Windows

This is my second part of the blog series on Using ELK Stack with BizTalk where I am going to write about notes on installing ELK stacks in Windows. In the first part I blogged about Part 1: Introduction about ELK Stack

Note: My presentation on “Using ELK stack with BizTalk Server” for Integration-Monday can be viewed here http://www.integrationusergroup.com/using-elk-stack-biztalk-server/

Download the following required software and tools:

Elasticsearch v5.4.1: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.1.zip

Logstash v5.4.1: https://artifacts.elastic.co/downloads/logstash/logstash-5.4.1.zip

Kibana v5.4.1: https://artifacts.elastic.co/downloads/kibana/kibana-5.4.1-windows-x86.zip

NSSM (To make “Logstash” and “Kibana” run as windows service): https://nssm.cc/release/nssm-2.24.zip

Unzip all the zip files to a folder – “C:\MRASHWINPRABHU\Projects\ELKStack”. So in this blog, I’ll execute commands from this path.

(Folder structure mentioned here are just for reference purpose. I use this folder path for my installation. So for your purpose, ensure you specify the folder path where you have downloaded the above mentioned software and tools)

Server Prerequisites:

Download and install the JDK:

Elasticsearch and Logstash require Java, so download the latest version from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

  • Accept License Agreement from “Java SE Development Kit Vxx” (Vxx is the build version, this changes frequently so get the latest version available)
  • Download the “Windows x64 (jdk- Vxx -windows-x64.exe)” package. (Again built version changes, so download the latest version available).
  • Install the package with default options.

Add the JAVA_HOME Environment variable:

Once the JDK is installed, you need to add the JAVA_HOME  Environment variable on the server:

Goto Control Panel  –> System and Security –> System. And click on the “Change settings”

Environment Variable Step1

Select “Advanced” tab –> Click on “Environment Variables…”

Environment Variable Step2

In the “Environment Variables” windows, Click on “New”

Environment Variable Step3

In the “New User Variable” window, fill in the values as following and click “OK”

Environment Variable Step4

Variable Name: JAVA_HOME

Variable value: C:\Program Files\Java\jdk1.8.0_131

Note: At the time of this blog, JDK version 1.8.0_131 is the latest. Choose the latest version of JDK you have installed in the above mentioned steps.

Install IIS:

Ensure IIS is installed on your Windows Server. I am not going to go through the steps to install IIS as this is standard. But ensure you have the following options, select as part of the IIS installation in the “Add Roles and Features Wizard”

Add Roles and Features Wizard

Add “Application Request Routing”:

Download and install “Microsoft Web Platform Installer 5.0” from https://www.microsoft.com/web/downloads/platform.aspx.

Once installed in the Web Platform Installer 5.0, in the search box at the left corner of this window, type “ARR”. In the list ensure “Application Request Routing 3.0” is added

Web Platform Installation 5.0

Install URL Rewrite:

In the search box of the Web Platform Installer 5.0, search for “URL Rewrite” and install the “URL Rewrite 2.0”

Web Platform Installation 5.0 -2

Once installed, open the IIS and you shall see the windows as :

IIS

Configure Kibana in IIS:

In IIS’s left panel, right click on “Sites” and select “Add Websites…”

In the “Add Websites” windows configure the value as show replacing it to your Site name, IP address, physical path and host name.

ELK-Part2-Add Wesite Properties

Update the host file:

Go to C:\Windows\System32\drivers\etc folder, open the hosts file and at the last line of the page add the value

127.0.0.1 loghost.fortuvis.net

ELK-Part2-Host File

Set Reverse proxy in IIS for Kibana:

In IIS, we configured Kibana to listen on localhost, we must set up a reverse proxy to allow external access to it.

Click on your newly created website name in the left panel of the IIS. For the selected website, on the right panel, double click on “URL Rewrite”.

On the URL Rewrite panel, click on “Add Rules”

Select “Reverse Proxy” under “Inbound and Outbound Rules” section.

Then you will receive a popup for “Add Reverse Proxy Rules”, click “OK” button. (Understand the mentioned warning before clicking the “OK” button)

Reverse Proxy Rules

Fill in the reverse proxy rules as follows:

Reverse Proxy Rules-2

IP Address “127.0.0.1” and port “5601” this is where Kibana is hosted.

As of now, we have configured the server with JAVA, IIS, Kibana and reverse proxy for Kibana.

Now it’s time to configure the ELK Stack:

Configure Elasticseach:

Open powershell and type the following commands

Navigate to the elasticseach’s bin folder: cd D:\ELK\elasticsearch\bin

Run command: ./elasticsearch-service manager

Elasticsearch Service Manager

Now you can configure the Elasticsearch to run as a service. And to configure Elasticsearch service to run on server boot, from the popped up service properties window, set the “Startup type” as “Automatic” and press “Apply”.

Ensure Service Status is “Started”, otherwise click “Start” button and click “Ok”.

Now you have configured Elasticsearch. To test the status of the Elasticsearch, open a web browser and type “127.0.0.1:9200”.

You shall see a json message like the following which ensures that the Elasticseach is configured and started. I chose to run 127.0.0.1:9200 from chrome browser as this browser by default can display the JSON content without any change in the browser settings.

Elasticsearch Browse

Install and Configure Logstash:

Logstash requires a config which can be downloaded from https://raw.githubusercontent.com/ulyaoth/tutorials/master/configs/windows/logstash/logstash.conf

Save this config file in your Logstash folder.

Now we will use downloaded NSSM to control the installation and configuration of the Logstash.

Open the Powershell –> Navigate to the NSSM’s “win64” folder –> type “.\nssm.exe install Logstash”

This will open the NSSM user interface where you can install Logstash and create it to run as a service.

NSSM Install Logstash

In the UI, configure the following in the “Application” tab:

Path: C:\MRASHWINPRABHU\Projects\ELKStack\logstash-2.3.1\ logstash-2.3.1\bin\logstash.bat

Startup directory: C:\MRASHWINPRABHU\Projects\ELKStack\logstash-2.3.1\ logstash-2.3.1\bin

Arguments: -f C:\MRASHWINPRABHU\Projects\ELKStack\logstash-2.3.1\ logstash-2.3.1\bin\logstash.json

Move to “Details” tab and fill in the values as shown:

Logstash Details Config

Move to the “Dependencies” tab and add the following: elasticsearch­service­x64 . This step ensures that Elasticsearch service is running for Logstash.

Now press “Install service” button to finish.

Install and configure Kibana:

Again, we will use NSSM to install Kibana as a service.

Open the Powershell à Navigate to the NSSM’s “win64” folderà type “.\nssm.exe install Kibana”

This will open the NSSM user interface where you can install Kibana and create it to run as a service.

In the UI, configure the following in the “Application” tab:

Path: C:\MRASHWINPRABHU\Projects\ELKStack\kibana\bin\kibana.bat

Startup directory: C:\MRASHWINPRABHU\Projects\ELKStack\kibana\bin

Arguments:

Move to the “Details” tab and fill in the value as:

Kibana Details Config

Move to the “Dependencies” tab and add the following:

elasticsearch­service­x64

Logstash

This step ensures that Elasticsearch service is running for Logstash.

Kibana Dependencies Config

Now press “Install service” button to finish.

We have installed ELK (Elasticsearch, Logstash and Kibana). Now go services tray and ensure following service are started:

  • Elasticsearch
  • Logstash
  • Kibana

Setup Kibana:

If above steps are followed correctly, you shall see the following when you browse to http://127.0.0.1:5601. Note: 127.0.0.1:5601 is the IP where Kibana is hosted.

Kibana Page

As you can see, you will see greyed out “Unable to fetch mapping, Do you have indices matching the pattern?” button. Because there are no index defined yet. To use Kibana, we must configure at least one index pattern. Index patterns are used to identify the Elasticseach index to run search and analytics against.

In the next part of this blog series, we will configure Logstash config file to capture data from multiple source, involve BizTalk to provide logs for on those sources, define indexes and search them in Kibana.

Posted in: BizTalk, Elasticsearch, ELK Stack, Kibana, Logging, Logstash Leave a comment June 20, 2017

About M.R.ASHWINPRABHU

M.R.ASHWINPRABHU is the founder and CEO of Fortuvis Systems Limited, a consulting company specialised in Microsoft technologies. Ashwin is a highly experienced integration consultant who works with clients to deliver high quality solutions. He works as technical lead developer, application architect and consultant, specializing in custom applications, enterprise application integration (BizTalk), Web services and Windows Azure.

Leave a Reply

Your email address will not be published. Required fields are marked *