After concentrating on learning Salt over the last few weeks I have been constantly surprised by how simple yet powerful the system is. When I first began, I started by setting up my base configurations (users, iptables, ssh, etc) into states and as I progressed things began getting more and more sophisticated.
For the most part everything has been completely smooth but as with learning anything new I have gotten stuck from time to time trying to figure out how to do something or troubleshooting a formatting error I introduced. Luckily with such a great and active community as well as great documentation these rough spots have been few and far between.
With this state I hope to go over some of these rough spots that I ran into, in the hope that it will help other people. The following state will:
- Transfer the OSSEC software to a minion
- Compile and install OSSEC using defined settings from pillar
- Thanks to daviddyball‘s example which really helped me get started
- Start up the ossec-authd service on the master using Saltstack reactor / event system
- Start up the ossec-authd service on the agent
- Negotiate keys between the master and agent
- Shut down the ossec-authd service on the master using Saltstack reactor / event system
- Start the OSSEC services
For this setup the config will look like this (along with a change to /etc/salt/master):
/srv/reactor - ossec.sls /srv/pillar - top.sls - servers.sls - packages.sls /srv/pillar/service-ossec - init.sls /srv/pillar/service-openssl - init.sls /srv/salt - top.sls /srv/salt/linux-server/ossec - init.sls - agent.sls - server.sls - preloaded-vars.jinja - local_rules.xml /srv/salt/linux-server/ossec/files - ossec-hids-2.7.tar.gz /srv/salt/linux-server/build-utils - init.sls - ssldev.sls
Reactor / Event System
To begin breaking this down I figured I would start with one aspect that took me a little while to get a grasp on, reactor and the event system.
In order to negotiate keys automatically, OSSEC has a service called ossec-authd that can be started from the server and the client to automatically generate and distribute keys. The only problem is it will generate/distribute a key for any ossec-authd request so the OSSEC manual recommends you do not leave it running all the time.
After some research I figured this would be a good way to try out the reactor and event system. So, to begin using reactor I added the following to my /etc/salt/master file:
reactor: - 'ossec': - /srv/reactor/ossec.sls
The above configuration will listen for any event that is tagged as “ossec” and will “react” by running the commands in the ossec.sls file which can be configured in much the same way as a state.
All events will have a tag and data associated with it (as well as other things such as the minion name). According to the Saltstack documentation “A “tag” allows for events to be filtered.” which makes parsing large amounts of event data more efficient. In order to see these events and the data they contain Saltstack has provided a python script that can be run to watch for events. This was invaluable to me when setting up my event:
import salt.utils.event event = salt.utils.event.MasterEvent('/var/run/salt/master') for data in event.iter_events(tag='auth'): print(data)
We now have reactor started and listening for events tagged as ossec but we need to setup our ossec.sls file in order to start doing something with these events. The following .sls file will look at the data of the event and perform different commands on our ossec server based on the data. Ultimately I would like to define the tgt and the path using pillar but as of this writing I cant get it working with pillar.
{% if data['data'] == 'ossec-auth-start' %} ossec-auth-start: cmd.cmd.run: - tgt: ubuntu01 - arg: - /var/ossec/bin/ossec-authd -p 1515 >/dev/null 2>&1 & {% elif data['data'] == 'ossec-auth-stop' %} ossec-auth-stop: cmd.cmd.run: - tgt: ubuntu01 - arg: - pkill ossec-authd >/dev/null 2>&1 {% endif %}
I am getting ahead of myself but in order to have a complete example I thought it would be useful to see. Later, in the state the client will execute the following salt-call in order to fire an event with the ossec tag and trigger the reactor:
salt-call event.fire_master 'ossec-auth-start' 'ossec'
Pillar
For my configuration I am trying to design as much as possible around pillar. I have centralized configuration files for my servers and desktops where I can define anything that is system dependent (IP address, what services are running, etc) and then I have service-<SERVICENAME> pillars to define anything that is application specific (paths, ports, uid/gid, etc).
To begin I will define my service-ossec pillar by creating a new directory called /srv/pillar/service-ossec and creating an init.sls inside of the directory with the following values:
service-ossec: version: '2.7' {# version of software to be installed (taken off of .tar file) #} language: 'en' userdir: '/var/ossec' {# Installation path #} en_active_response: 'y' {# Enable active response #} en_syscheck: 'y' {# Enable system checks #} en_rootcheck: 'y' {# Enable rootkit detection #} en_update_rules: 'y' {# Update rules #} en_syslog: 'y' {# Enable syslog checks #} server_ip: '172.16.0.10' {# IP of the OSSEC master server #} en_email: 'y' {# Enable email notifications #} email_address: '[email protected]' {# Email address to send notifications #} white-list: '172.16.0.0/16' {# IP whitelist #}
Since I also use openssl for a variety of services in my environment I have a centralized service-openssl pillar defined in /srv/pillar/service-openssl/init.sls. These values are then used and shared between multiple programs in my environment. One of which is OSSEC.
service-openssl: defaultbits: 2048 countrynamedefault: US stateorprovincenamedefault: California orgnamedefault: My Company Name
Next, I use a centralized packages.sls pillar so I can deal with differing package names between Debian and RedHat based systems. While this list appears to be growing, here is what I have now:
packages: {% if grains['os_family'] == 'Debian' %} apache: apache2 php: libapache2-mod-php5 git: git-core snmp: snmpd snmp-service: snmpd ssh: ssh ssh-service: ssh vi: vim ntp: ntp ntp-service: ntp python-mysql: python-mysqldb mysql-service: mysql winbind: winbind samba-service: smbd build-env: build-essential ssl-dev: libssl-dev auth-pam: libapache2-mod-auth-pam {% elif grains['os_family'] == 'RedHat' %} apache: httpd php: php git: git snmp: net-snmp snmp-service: snmpd ssh: openssh ssh-service: sshd vi: vim-enhanced ntp: ntp ntp-service: ntpd python-mysql: MySQL-python mysql-service: mysqld winbind: samba-winbind samba-service: smb build-env: make ssl-dev: openssl-devel auth-pam: mod_auth_pam {% endif %}
Now, I will update my existing servers.sls and define a server to be the OSSEC server and one to be the OSSEC agent. Please note that I have included more information in this example to help detail how I am using this .sls file but for the purposes of this example all I really need to define here is the roles ossec.agent and ossec.server:
servers: ubuntu01: roles: - network - ssh - ossec.server - users - iptables - apache - owncloud websites: - owncloud tcp_ports: - 80 - 443 domain: domain.local gateway: 172.16.0.1 gwdev: eth0 dns: - 172.16.0.10 - 172.16.0.11 network_adapters: eth0: { en: 'True', ip: 172.16.0.10, sn: 255.255.0.0, nw: 172.16.0.0, bc: 172.16.255.255 } eth0:1: { en: 'True', ip: 172.16.0.100, sn: 255.255.0.0, nw: 172.16.0.0, bc: 172.16.255.255 } centos01: roles: - network - users - iptables - ossec.agent tcp_ports: - 80 - 443 domain: domain.local gateway: 172.16.0.1 gwdev: eth0 dns: - 172.16.0.10 - 172.16.0.11 network_adapters: eth0: { en: 'True', ip: 172.16.0.20, sn: 255.255.0.0, nw: 172.16.0.0, bc: 172.16.255.255 }
Lastly, now that both pillars are setup we need to make sure they are defined in the /srv/pillar/top.sls:
base: '*': - servers - packages - service-ossec - service-openssl
State – top.sls
While we are still close to the topic of how I setup my pillar’s I wanted to touch on how I setup my state top.sls file. My state top.sls file will loop through the roles defined in the servers.sls and apply the corresponding state based on its OS type using grains. So my /srv/salt/top.sls file looks like this:
{% set hostname=grains['id'] %} {% if grains['kernel'] == 'Linux' %} {% set kernel='linux' %} {% elif grains['kernel'] == 'SunOS' %} {% set kernel='solaris' %} {% elif grains['kernel'] == 'Windows' %} {% set kernel='windows' %} {% elif grains['kernel'] == 'Darwin' %} {% set kernel='macintosh' %} {% endif %} base: '*': - common.salt 'os_family:Debian': - match: grain - common.apt {% if pillar['servers'][hostname] is defined %} {% for roles in pillar['servers'][hostname]['roles'] %} 'servers:{{ hostname }}:roles:{{ roles }}': - match: pillar - {{ kernel }}-server.{{ roles }} {% endfor %} {% endif %} {% if pillar['desktops'][hostname] is defined %} {% for roles in pillar['desktops'][hostname]['roles'] %} 'desktops:{{ hostname }}:roles:{{ roles }}': - match: pillar - {{ kernel }}-desktop.{{ roles }} {% endfor %} {% endif %}
When the top.sls file parses through my servers.sls file it will find the role ossec.server and ossec.agent and apply those states to the servers ubuntu01 and centos01. Since both of these servers are linux servers it will apply the ossec state from the linux-server directory.
State – linux-server.ossec
In order to create my OSSEC state I have created the following directories to hold my state files /srv/salt/linux-server/ossec and /srv/salt/linux-server/ossec/files. My state begins with the init.sls file which will:
- Install compilation prerequisites from two other states (defined below)
- Copy the ossec tarball to the server from the /srv/salt/linux-server/ossec/files directory
- Decompress the tarball
- Copy the answers to the installer prompts to the ossec installation directory
- Run the installer
In order to manage the state centrally I have defined all of the options in the service-ossec pillar we created earlier.
{% set hostname = grains['id'] %} {% set version = pillar['service-ossec']['version'] %} {% set ossecdir = 'ossec-hids-{0}'.format(version) %} include: - linux-server.build-utils - linux-server.build-utils.ssldev ossec-install-directory: file.directory: - name: /usr/src/ossec-install - order: 500 ossec-download-installer: file.managed: - source: salt://linux-server/ossec/files/{{ ossecdir }}.tar.gz - name: /usr/src/ossec-install/{{ ossecdir }}.tar.gz - order: 501 - require: - file: ossec-install-directory ossec-extract-installer: cmd.run: - name: 'tar -zxf {{ ossecdir }}.tar.gz' - cwd: '/usr/src/ossec-install/' - unless: stat {{ pillar['service-ossec']['userdir'] }}/bin/ossec-control - order: 502 - watch: - file: ossec-download-installer ossec-installer-variables: file.managed: - name: '/usr/src/ossec-install/{{ ossecdir }}/etc/preloaded-vars.conf' - source: 'salt://linux-server/ossec/preloaded-vars.jinja' - template: jinja - order: 504 - watch: - cmd: ossec-extract-installer ossec-install: cmd.run: - name: '/usr/src/ossec-install/{{ ossecdir }}/install.sh' - unless: stat {{ pillar['service-ossec']['userdir'] }}/bin/ossec-control - require: - pkg: make-package - pkg: gcc-package - pkg: ssldev-package - order: 505 - watch: - file: ossec-installer-variables
State – linux-server.ossec.server
Next, I want to be able to manage both by OSSEC server and agent using the same configuration so I created a server.sls file which will perform a server install.
The sls:
- Manages my OSSEC local_rules.xml file
- Generates an SSL key and certificate which is used by ossec-authd
- Starts the OSSEC service
- Creates a cron job to ensure that ossec-authd is not running for very long in the case it is not shut down properly.
{% set hostname=grains['id'] %} {% set domain=pillar['servers'][hostname]['domain'] %} include: - linux-server.ossec rules-config: file.managed: - name: {{ pillar['service-ossec']['userdir'] }}/rules/local_rules.xml - user: root - group: ossec - mode: 550 - order: 510 - source: salt://linux-server/ossec/local_rules.xml ssl-key: cmd.run: - name: openssl genrsa -out {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.key 2048 - unless: stat {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.key - order: 510 ssl-cert: cmd.run: - name: openssl req -subj '/CN={{ hostname }}.{{ domain -}} /C={{ pillar['service-openssl']['countrynamedefault'] -}} /ST={{ pillar['service-openssl']['stateorprovincenamedefault'] -}} /O={{ pillar['service-openssl']['orgnamedefault'] }}' -new -x509 -key {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.key -out {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.cert -days 730 - unless: stat {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.cert - onlyif: stat {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.key - order: 510 ossec-service: service.running: - name: ossec - enable: True - sig: ossec-syscheckd - order: 511 - require: - cmd.run: ossec-install {# Since OSSEC does not recommend keeping authd running my workaround so far is to run a cron job to shut it off after a period of time. I need to figure out a better way of doing this in the future as this may require more than one highstate to add keys for a client and it is cheesy #} pkill ossec-authd >/dev/null 2>&1: cron.present: - user: root - minute: '*/5'
State – linux-server.ossec.agent
As with the server example above, the following config sets up our agents. This part of the configuration is where we start using the event system and reactor.
The agent.sls:
- Fires an event to the salt master from the minion which will signal reactor to start the ossec-authd service on the OSSEC server
- The OSSEC agent then starts the ossec-authd service and negotiates keys with the OSSEC server
- After the negotiation is done the minion fires a new event to the salt master which will signal reactor to shut down the ossec-authd service on the OSSEC server.
- Finally the OSSEC services will be started on the agent.
include: - linux-server.ossec {# Use events/reactor system to start up the ossec-authd process on the OSSEC master #} server-auth: cmd.run: - name: salt-call event.fire_master 'ossec-auth-start' 'ossec' - unless: stat {{ pillar['service-ossec']['userdir'] }}/etc/client.keys - order: 510 {# OSSEC authd agent connects to master and registers its key #} agent-auth: cmd.wait: - name: sleep 1 && {{ pillar['service-ossec']['userdir'] }}/bin/agent-auth -m {{ pillar['service-ossec']['server_ip'] }} -p 1515 - unless: stat {{ pillar['service-ossec']['userdir'] }}/etc/client.keys - order: 511 - watch: - cmd.run: server-auth {# We are done creating our key so lets shut down the ossec-auth process on the master using reactor #} server-auth-shutdown: cmd.wait: - name: salt-call event.fire_master 'ossec-auth-stop' 'ossec' - order: 512 - watch: - cmd.wait: agent-auth {# Start the OSSEC services on the agent #} ossec-service: service.running: - name: ossec - enable: True - sig: ossec-syscheckd - order: 513 - require: - cmd.run: agent-auth
Config File – preloaded-vars.jinja
The preloaded-vars.jinja is used by the OSSEC installation to automatically answer any questions during the install. For our configuration everything is defined in the service-ossec pillar and the options that are required for a server only show up if you have defined the ossec.server role in the servers.sls.
{%- set hostname=grains['id'] %} USER_LANGUAGE="{{ pillar['service-ossec']['language'] }}" USER_NO_STOP="y" {%- if 'ossec.server' in pillar['servers'][hostname]['roles'] %} USER_INSTALL_TYPE="server" USER_EMAIL_SMTP="{{ pillar['resources']['smtp'] }}" {%- else %} USER_INSTALL_TYPE="agent" USER_AGENT_SERVER_IP="{{ pillar['service-ossec']['server_ip'] }}" {%- endif %} USER_DIR="{{ pillar['service-ossec']['userdir'] }}" USER_DELETE_DIR="y" USER_ENABLE_ACTIVE_RESPONSE="{{ pillar['service-ossec']['en_active_response'] }}" USER_ENABLE_SYSCHECK="{{ pillar['service-ossec']['en_syscheck'] }}" USER_ENABLE_ROOTCHECK="{{ pillar['service-ossec']['en_rootcheck'] }}" USER_UPDATE_RULES="{{ pillar['service-ossec']['en_update_rules'] }}" USER_ENABLE_EMAIL="{{ pillar['service-ossec']['en_email'] }}" USER_EMAIL_ADDRESS="{{ pillar['service-ossec']['email_address'] }}" USER_ENABLE_SYSLOG="{{ pillar['service-ossec']['en_syslog'] }}" USER_WHITE_LIST="{{ pillar['service-ossec']['white-list'] }}"
local_rules.xml and files/ossec-hids-2.7.tar.gz
From your OSSEC installation make a copy of the local_rules.xml file and put it in the /srv/salt/linux-server/ossec state directory so that you can manage the default OSSEC rules centrally.
At this point you will also need to download a copy of OSSEC and place it in the files directory to enable the OSSEC software installation. Please note the version number on this file as it must correspond with the version set in our service-ossec pillar.
Final dependencies
In order for OSSEC to properly install we need a compiler and the ssl-dev packages so that we can build the ossec-authd service. I used to define these in my OSSEC state but as I now have other installers that require the same thing I have moved these out to a state of their own under /srv/salt/linux-server/build-utils and I call them using includes.
Since this is pretty basic I am just going to include the contents of both files
init.sls:
make-package: pkg: - name: make - order: 50 - installed gcc-package: pkg: - name: gcc - order: 50 - installed
ssldev.sls:
ssldev-package: pkg: - name: {{ pillar['packages']['ssl-dev'] }} - order: 50 - installed
Final notes
I am working on putting together a full pillar / state environment to upload on to github to allow anyone to download a copy of my environment but please keep in mind the beauty of Saltstack is that it can be configured and reorganized in a variety of ways.
While this environment was built with the idea of centralized configuration using pillar I have also thought about changing things around to be a bit more dynamic. I guess that is part of the beauty of this software. It is extremely flexible.
I know I’m dredging up sohnmtieg old here. I have the latest version of ossec compiled on ubuntu 11.10. I can get an agent to authenticate no problems. I’m working in AWS right now and getting prepared for autoscaling certain portions of an application.I see no easy way to automatically delete agents that have been murdered due to low use from the application stand point. Where as when a new machine in a machine class spins up it will automatically add itself.What’s the best way to go about auto deletion when an instance is going the way of the do-do?
While I havent began working on something to do this yet the thought of building something has crossed my mind. To start, it looks like OSSEC keeps a master list of all the client keys in /var/ossec/etc/client.keys on the master. If this is the only place that this information is stored I am thinking that we could probably create a state using salt.states.file.sed to remove a line from this file where the hostname in the file is matched to a grain or something else.
Since autoscaling shuts down machines when they are not needed we could possibly have a script run from the machine that is being shutdown to trigger the state (probably using reactor). Hope this helps and would love to know how it turns out.
So how about that github repo with it that you mentioned? 😉
It would certainly make following the code easier.
I know, you are absolutely right! I will tell you what. I am at re:invent this week but when I get back I will see if I can get something together.
How is it proceeding? 😉
Sorry, been a little busy lately. I just uploaded some of the states to github. I tried to make sure that I caught all of my dependancies so they should work on 16.4 (Although I didnt have time to test them out). Also, just an FYI. Right now there is a bug with Jinja rendering on the 17.x releases which has broken some of these states.
Here is the link: https://github.com/gl1ch/salt-relational
Hope this helps!
Andrew
Aewsome, thanks!
This is great stuff.
Salt has progressed a bit – ordering is now implicit and you can drop all the ” – order: ” args.
OSSEC has two ways of setting up clientserver agent auth. The other way is via manage_agent.
I wrote about how I made a pillar for openvpn that automatically fetches or creates SSL certs for minions here: http://garthwaite.org/virtually-secure-with-openvpn-pillars-and-salt.html. Search for “/srv/pillar/openvpn.sls”
I plan to do the same for OSSEC. Blog post to follow.
https://github.com/saltstack/salt/issues/19020
you have a problem?
Thanks! Sorry for the late reply! Had a kid this year so it has been a little crazy keeping up with everything.
Andrew