Nagios and Puppetmaster with PuppetDB

admin — Sat, 20 Jul 2013 19:31:54 +0000

Nagios (3.5.1) is an open source tool for monitoring IT infrastructure. Coupled with Nagios, Puppet (3.7.3) can automatically configure machine profiles for each node of that infrastructure, allowing you to track and monitor all your puppetted machines. This devopera build features a configured Puppetmaster (puppet in a master/daemon role), setup to use PuppetDB for storing all machine profiles (storeconfigs). The Nagios admin interface is installed in /usr/share/nagios/html and its configuration is stored in /etc/nagios. There are symlinks in the web user's home directory making it straight forward to start customising the setup using either Samba or SSH.

Nagios admin interface

Dev/staging

Simply enter the IP address of your server or VM into the address bar of your web browser, using port 43326, e.g.
http://<ip_address>:43326

Live

Port 43326 is not exposed on live servers, but the admin interface can be accessed using an SSH tunnel.

Credentials

Dev/staging/live

The admin interface has some elementary password protection (htbasic), though this should not be relied upon outside of a dev/staging environment. User (admin) has an initial password 'admLn**'. Usernames and passwords are case sensitive.
Admin username: admin
Admin password: admLn**

LAMP stack

Zend Server (5.6.0)

Dev/staging/live

Zend make PHP. They also formally release a partial stack for hosting PHP applications like Drupal and Wordpress.

Apache (2.2)

Apache runs as user web and group www-data. The application site files are owned by the web user but given the web group to allow Apache read-only access to them. Certain content management systems (CMS) require that key subdirectories are writeable (files, cache etc.) but those exceptions can be set using chmod 660 <filename>. The generic config (/etc/httpd/conf/httpd.conf) is fairly vanilla; specialisations and vhosts live in /etc/httpd/conf.d/.

PHP (5.3)

This version of PHP has faster execution times and more efficient memory handling, support for closures and true anonymous functions.

Zend optimizer

By default Zend optimizer is setup to compile all PHP to accelerate its execution, much like APC. This can be configured in the Zend console

Zend debugger

You can connect your Integrated Development Environment (IDE) such as Eclipse or Komodo to the Zend debugger on port 10000. Depending on your firewall setup, you may choose to tunnel access to this service over SSH.

Zend console

Dev

The Zend server console runs on its own lightweight web server (Lighttpd) on port 10081, which is exposed through the firewall to allow direct external access to the console (using a web browser) or the debugging (using an IDE, such as Zend Studio or Eclipse).

Staging/Live

The console, like other firewalled services, can be accessed using SSH tunnels.

MySQL (5.5)

MySQL comes in its latest general release format, to ensure the best mix of performance and reliability. It's running on the standard port (3306 and can be accessed via a tunnelled connection or from the installed command line client (mysql).

Automated root access

The mysql root password is setup in /root/.my.cnf which allows the [linux] root user to login to the mysql command line automatically, without being re-prompted for a password.

Configuration

The generic config (/etc/my.cnf) is fairly vanilla; the main configuration lives in /etc/mysql.d/domysqldb.conf and is broadly divided into static values (consistent for all machine types and sizes) and dynamic values that alter depending on how much memory or how many processor cores the machine is built with.

Credentials

Dev/staging

The administrator account (root) has an initial password 'admLn**'. Usernames and passwords are case sensitive.
MySQL root username: root
MySQL root password: admLn**

Live

All initial usernames and passwords for our live builds are supplied to you by email at the time of delivery.

domysqldb

dozendserver

Operating base

Operating base (applies to CentOS and Ubuntu)

Single user

Each machine is setup with a single day-to-day user account (web) and a super-privileged root user. The web user's home directory is in /home/web.

Secure shell (SSH)

SSH access is setup on a non-standard port (15022) to improve security.

SSH access using password

Dev

web only, then su to root or better sudo

Staging/live

web only, then su to root

SSH access using PKI

Public-private key pairs give excellent security and are a handy way to manage communication, both to your server and from it to other sites, such as github.com. Every Devopera server and VM comes with a private-public key pair installed. By copying out the private key and loading it on your local host machine, you can SSH to your server without being prompted to login (instructions for Windows/Putty/Pageant, Linux and Mac OS X).

PKI key agent

Dev

The private key is loaded automatically using a hardcoded passphrase. This is very convenient for dev machines, because they can talk to other non-secure infrastructure as soon as started up.

Staging/live

Live and staging servers rely on the passphrase being entered the first time the web user logs on. From that point on it's stored by the ssh-agent (until the next restart or refresh of ssh-agent) and used to authenticate the machine with other servers. It does mean that live servers require the user to enter the key passphrase before they're able to start talking, but that additional security measure helps ensure they don't go rogue.

Credentials

Dev/staging

web user password: admLn**
root user password: devopera
Key passphrase: admLn**

Live

All initial usernames and passwords for our live builds are supplied to you by email at the time of delivery.

SFTP access

Secure File Transfer Protocol (SFTP) is also enabled by the SSH daemon. While almost all command-line versions of the SSH client support port (-p) and user@host settings, some older versions of SFTP don't. However you can configure all the settings (port, user, host, even forwarding) in ~/.ssh/config which simplifies your command line (ssh or sftp ).

Security-Enhanced Linux (SELinux)

SELinux is enabled and set to enforcing for both dev and live machine types. All services described here are configured to work with SELinux in their current configurations. While many sysadmins choose to disable SELinux, we prefer to start with the most secure setup we know how to create and let you make decisions about how you want to relax it. Please see later notes on how to maintain the security of your devopera servers.

Standard tools

NTP

Network Time Protocol is used to synchronise this machine's time with network-based time servers.

Compass

Dev/staging

Compass is an open-source CSS Authoring Framework, mainly used to compile SASS into CSS.

Live

Dev tools are not installed on live servers by default.

Version control systems (VCS)

Git/Subversion repos

Git and subversion are setup. Many applications are available from repositories. These applications are organised in the /var/www directory by the VCS they use and the server from which they are sourced:
e.g. /var/www/git/github.com/my_git_repo

Samba

Dev

Samba shares are setup for the web user's home directory and /var/www. Each samba share is configured to create and modify files with a certain set of permissions: read-write for web and read-only for www-data.

No matter how diligent your are, you may find certain programs set different permissions so a bit of chmod'ing is often required! To access samba using Windows file sharing, just connect to:
\\<ip address>
or
\\<machine alias>
if you've set one up.

Staging/live

We don't encourage using samba for access to live machines. While it can be enabled and port-forwarded over SSH securely, we recommend using a VCS such as git/svn for code deployment and SSH/SFTP or SCP to send files to the server.

Credentials

Dev

The web user account has a corresponding but independent Samba user (web) with initial password 'admLn**'. Usernames and passwords are case sensitive.

Samba username: web
Samba password: admLn**

Staging/Live

The Samba service is not turned on by default for live servers or their corresponding staging peers. It's for security.

Firewall

Dev

iptables

We restrict only the incoming port list to those services that are available. For dev machines, your environment may be properly secured behind a firewall, so you could disable the VM's firewall by running sudo service iptables stop

Staging/live

Live servers typically run in exposed environments or on open networks, hence our firewall policy is a lot more involved for staging VMs or live builds.

ConfigServer Firewall (csf) and Login Failure Daemon (lfd)

ConfigServer Firewall is setup to protect both incoming and outgoing ports. If you want to do install new services, you must open the necessary ports in /etc/csf/csf.conf and ask CSF to reload them.

rkhunter

RootKit hunter is designed to identify binaries that have been corrupted with intent to compromise a server. To run rkhunter manually (non-interactive):
/usr/bin/rkhunter -c -sk
After updating any of the monitored binaries, you'll need to tell rkhunter to update its database in order to avoid false positives:
/usr/bin/rkhunter --propupd
Finally, if you receive rkhunter's 'something has changed' email (that reads "Please inspect this machine, because it may be infected."), we recommend looking at the logs (in /var/log/rkhunter) to understand what triggered it.

maldet

Linux Malware Detect looks for malware on a server.

DNS

Dev

Currently dev machines do not offer local DNS support, but it is planned. For now, you can always put records in your /etc/hosts file, which under Windows lives in /Windows/system32/drivers/etc/hosts.

Staging/live

DNS is not configured for live servers as the domain name resolution system was not really designed for web servers to host their own DNS records. We recommend using a registrar that provides DNS management services, like Domainmonster.com who we use.

General information

General VM information

For more information on how to use the virtual application (VM) you've downloaded:
Using a Devopera VM

Download a free CentOS 6 development VM to get started with Nagios and Puppetmaster with PuppetDB right away
doco6-nag3r004-dev.tar.gz (1459.1MB)

Let us configure your Nagios and Puppetmaster with PuppetDB server for £0

CentOS 6(active tab)

Download Now CentOS 6 VM (doco6-nag3-dev.tar.gz)

Devopera VMs are Vagrant-compatible base boxes. To download from the command line:

...

A Puppet to many puppetmasters

admin — Sun, 07 Jul 2013 16:53:29 +0000

Puppet enables you to create a virtualised cloned infrastructure of your live systems. While Virtual machine (VM) technology is an important component of that, it's Puppet's ability to automatically build new machines based on complex manifests that makes a cloned instrastructure possible.

Puppeting the puppetmaster

All the state that makes up a puppetmaster is itself definable in a set of manifests and storable in a repository. There are a few complications to overwriting Puppet's main configuration directory (/etc/puppet) but they're handled by our open source dopuppetmaster module. Once the puppetmaster can be managed like all other nodes in the connected graph that makes up a complex infrastructure, complete isolated clones can be made to enable true test-as-live testing.

Storing the configs

The puppet master machine can be configured to store information about all the agents that connect to it. That information can then be used to configure host state.

Monitoring

One of the use cases enabled by stored configs is monitoring, using a tool such as Nagios. Each of the agent machines connects to the puppetmaster, retrieving its catalogue and storing its facts (from facter). By then re-puppeting the puppetmaster, effectively from itself, the puppetmaster can dynamically generate Nagios host config.

Modules:

dopuppetmaster

donagios

Applications:

Nagios and Puppetmaster with PuppetDB

devopera - dopuppetmaster

Nagios and Puppetmaster with PuppetDB

Nagios admin interface

Dev/staging

Live

Credentials

Dev/staging/live

LAMP stack

Dev/staging/live

Apache (2.2)

PHP (5.3)

Zend optimizer

Zend debugger

Zend console

Dev

Staging/Live

Automated root access

Configuration

Credentials

Dev/staging

Live

Operating base

SSH access using password

Dev

Staging/live

SSH access using PKI

PKI key agent

Dev

Staging/live

Credentials

Dev/staging

Live

SFTP access

NTP

Compass

Dev/staging

Live

Git/Subversion repos

Dev

Staging/live

Credentials

Dev

Staging/Live

Dev

Staging/live

Dev

Staging/live

General information

A Puppet to many puppetmasters

Puppeting the puppetmaster

Storing the configs

Monitoring