Note

This is a lightweight documentation intended to get operators started with setting up the Zuul service. For more insight on what Zuul can do, please refer to upstream documentation.

Operate zuul

  • The configuration is located in /etc/zuul
  • The logs are written to /var/log/zuul

By default, no merger are being deployed because the executor service can perform merge task. However, a merger can also be deployed to speed up start time when there are many projects defined.

Jobs default nodeset

The default configuration in /etc/software-factory/sfconfig.yaml for zuul nodeset is to use the label pod-centos-7. This label is only available if you added the role hypervisor-k1s in /etc/software-factory/arch.yaml. If you don’t use this role, you should specify the nodeset to use for jobs. For example, if you have defined a cloud image in nodepool configuration, you should update /etc/software-factory/sfconfig.yaml to specify the default nodeset name and label, for instance:

zuul:
  default_nodeset_name: cloud-centos-7
  default_nodeset_label: cloud-centos-7

Then, run sfconfig to apply the modification

Save and restore the queues

The zuul scheduler service is stateless and stopping the process will lose track of running jobs. However the zuul-changes.py utility can be used to save and restore the current state:

# Print and save all builds in progress to /var/lib/zuul/zuul-queues-dump.sh
/usr/libexec/software-factory/zuul-changes.py dump

systemctl restart zuul-scheduler

# Reload the previous state
/usr/libexec/software-factory/zuul-changes.py load

The periodic and post pipelines are not dumped by this tool.

Restart Zuul services

The zuul_restart.yml playbook stops and restarts Zuul services and automatically restore the scheduler’s jobs queues.

ansible-playbook /var/lib/software-factory/ansible/zuul_restart.yml

Configure an external gerrit (use Software Factory as a Third-Party CI)

Refer to the Third-Party-CI Quick Start guide

Add a git connection

In /etc/software-factory/sfconfig.yaml add in git_connections:

- name: gerrithub
  baseurl: https://review.gerrithub.io

Then run sfconfig to apply the configuration.

Create a GitHub app

To create a GitHub app on my-org follow this github documentation:

  • Open the App creation form:
    • to create the app under an organization, go to https://github.com/organizations/<organization>/settings/apps/new
    • to create the app under a user account, go to https://github.com/settings/apps/new
  • Set GitHub App name to “my-org-zuul”
  • Set Homepage URL to “https://fqdn”
  • Set Setup URL to “https://fqdn/docs/user/zuul_user.html”
  • Set Webhook URL to “https://fqdn/zuul/api/connection/github.com/payload”
  • Create a Webhook secret
  • Set permissions:
    • Repository Administraion: Read (get branch protection status)
    • Repository contents: Read & Write (write to let zuul merge change)
    • Issues: Read & Write
    • Pull requests: Read & Write
    • Commit statuses: Read & Write
  • Set events subscription:
    • Commit comment
    • Create
    • Push
    • Release
    • Issue comment
    • Issues
    • Label
    • Pull request
    • Pull request review
    • Pull request review comment
    • Status
  • Set Where can this GitHub App be installed to “Any account”
  • Create the App
  • In the ‘General’ tab generate a Private key for your application, and download the key to a secure location

To configure the Github connection in sfconfig.yaml, add to the github_connections section:

- name: "github.com"
  webhook_token: XXXX # The Webhook secret defined earlier
  app_id: 42 # The ID shown in the about section of the app.
  app_key: /etc/software-factory/github.key # Path to the private key generated during the setup of the app.
  app_name: app-name
  label_name: mergeit # Label of the tag that must be set to let Zuul trigger the gate pipeline.

Then run sfconfig to apply the configuration. And finally verify in the ‘Advanced’ tab that the Ping payload works (green tick and 200 response). Click “Redeliver” if needed.

Note

It’s recommended to use a GitHub app instead of manual webhook. When using manual webhook, set the api_token instead of the app_id and app_key. Manual webhook documentation is still TBD…

Check out the Zuul GitHub App user documentation to start using the application.

More information about the Zuul’s Github driver can be found in the Zuul Github driver manual.

Use openstack-infra/zuul-jobs

The zuul-scheduler can automatically import all the jobs defined in the zuul-ci.org/zuul-jobs repository. Set the zuul.upstream_zuul_jobs options to True in sfconfig.yaml

Restarting a config-update job

When the config-update job fails, you can manually restart the job using the command bellow. Make sure to set the ref-sha which is the last commit hash of the config repository.

zuul enqueue-ref --trigger gerrit --tenant local --pipeline post --project config --ref master --newrev ref-sha

The job will be running in the post pipeline of the Zuul status page.

Troubleshooting non starting jobs

  • First check that the project is defined in /etc/zuul/main.yaml
  • Then check in scheduler.log that it correctly requested a node and submitted a job to the executor
  • When zuul reports PRE_FAILURE or POST_FAILURE, then the executor’s debugging needs to be turned on
  • Finally passing all loggers’ level to DEBUG in /etc/zuul/scheduler-logging.yaml then restarting the service zuul-scheduler might help to debug.

Troubleshooting the executor

First you need to enable the executor’s keepjob option so that ansible logs are available on dist:

/usr/bin/zuul-executor keep
/usr/bin/zuul-executor verbose

Then next job execution will be available in /var/lib/zuul/builds/.

In particular, the work/ansible/job-logs.txt usually tells why a job failed.

When done with debugging, deactivate the keepjob option by running:

/usr/bin/zuul-executor nokeep
/usr/bin/zuul-executor unverbose

Accessing test resources on failure (autohold)

See the nodepool operator documentation.