ELK service

Software Factory bundles an ELK stack based on Opendistro for Elasticsearch to ease searching through the logs artifacts of jobs. Once activated, the console log of every build is exported through logstash and then searchable via Kibana.

A Software Factory user might want to export more artifacts than the job’s console. Indeed a job may generate additional log files. In that case a custom zuul post-run job must be defined. In order to do so a user must refer to Export logs artifacts to logstash.

How to activate

These services are not deployed by default but can be activated by adding the following components in /etc/software-factory/arch.yaml:

- elasticsearch
- logstash
- job-logs-gearman-client
- job-logs-gearman-worker
- kibana

Then running:

# sfconfig

The Kibana interface should be accessible via the Software Factory top menu under the name Kibana.

Manual actions after installing Software Factory

After the installation of Software Factory, or the activation of ELK, you must connect to Kibana as the admin user and go to the section “Discover” in Kibana menu to ensure that Kibana has created the index pattern. This is a per-requisite before a regular Kibana user (which has read-only access) can search through the jobs logs.

Note

After the first CI job execution, The job console logs are sent to ElasticSearch through Logstash. Then Kibana will be able to create the index pattern when connected as admin.

The password for admin is set as a elasticsearch_password entry in secrets.yaml file.

You can also use below command:

awk '/elasticsearch_password/ {print $2}' /var/lib/software-factory/bootstrap-data/secrets.yaml

Managing internal users

There are few users created in Elasticsearch in Software Factory. The full list of available users can be seen here

Users information are stored in internal_users.yml file which is located in opendistro_security config directory:

/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/

For each user, password has been generated by sfconfig tool. Not encrypted passwords are stored in:

/var/lib/software-factory/bootstrap-data/secrets.yaml

More information about the file, you can find here.

Available users

There are few users created in Software Factory for Elasticsearch:

  • admin - the superuser in Kibana. It has all permissions to manage the Kibana and Elasticsearch cluster
  • kibanaserver - this user is used by Kibana service to connect to the Elasticsearch
  • logstash - dedicated user to communicate logstash service to the Elasticsearch
  • curator - user that is used by curator service to ‘clean-up’ the index
  • kibana - a read-only user. This user shows on the login page

For each user, password has been generated by sfconfig tool. You can find them in:

/var/lib/software-factory/bootstrap-data/secrets.yaml

Resetting password

The best way to change the user password is to change it in the secrets.yaml file and re-run sfconfig tool.

To do that without the sfconfig tool, you need to generate it by using this script:

/usr/share/elasticsearch/plugins/opendistro_security/tools/hash.sh -p <MY PASSWORD>

The generated hash should be added into the internal_users.yml file as the “hash” property for the user in opendistro_security config directory.

Note

To apply that change, you need to run securityadmin.sh script. Don’t forget to put unencrypted password into the secrets.yaml file before executing sfconfig tool!

securityadmin.sh script

To apply changes, you need to execute the securityadmin script. It is the most important step that you should not forget after doing some changes.

In the Software Factory, the command below will setup the securityadmin.sh script. What you need to do is to execute it on the Elasticsearch host.

/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh   \
  -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/  \
  -icl -nhnv -cacert /etc/elasticsearch/certs/localCA.pem  \
  -cert /etc/elasticsearch/certs/elasticsearch-admin.crt  \
  -key /etc/elasticsearch/certs/elasticsearch-admin.key    \
  -h $(hostname)

Tenants

Software Factory is only configuring one tenant: global. Multi-tenancy is disabled (check Kibana configuration file).

Manage indices

Query the list and usage of index using:

curl http://elasticsearch:9200/_cat/indices?v

Delete old/unused index using:

curl -X DELETE http://elasticsearch:9200/INDEX-NAME