Can I replace a local Prometheus with AWS CloudWatch? And include syslog monitoring as well? Yes, I can. But it was a PITA to set up. Here’s how.
Why CloudWatch?
To learn, of course! Well, that and a practical test. It’s an experiment. I’ve run a local server for years and had it be my syslog server and handle my monitoring. Yes, I’m a geek. I use Ubiquity Unifi equipment and I have the controller configured to remote syslog so I get all the AP events. I also run my own internal DHCP for static mapping of MAC to IP, and internal DNS and Reverse DNS. I used to have DHCP/DNS on a different small server.
But running these servers costs me money, in electrical costs. I decided to see if could get comparable or better functionality using CloudWatch, and then compare the costs too. This post will be just about the first steps of setting things up. Later I’ll blog about more details on the service, and evaluate the costs. At this point even if it’s more money I’m going through this exercise, since I really want to understand how this would work.
Of course, to really evaluate the two options I’ll apply the 5 pillars of the Well Architected Framework - CORPS in short (Cost, Operational Excellence, Reliability, Performance, Security). But that’s another blog post. Today is just getting the agent set up!
CloudWatch Agent
You would think this is an easy slam dunk. AWS has a document describing how to do this. But while it says you can download it using SystemManager or an S3 link, the docs don’t actually include a link. And it’s very much written from the perspective that you will use SystemManager.
But I’m not. All I want to do is install the Agent locally, configure it, and have it send it’s logs and metrics off to CloudWatch. Turns out that was fraught with difficulty.
Google searches mostly turned up more on how to do it with SystemManager, or how to do it on an EC2 instance. I did find this post that was groundbreaking for me. But it had some errors and misses too.
Installation
The [agent debian package ](https://s3.amazonaws.com/amazoncloudwatch-agent
/debian/amd64/latest/amazon-cloudwatch-agent.deb) is installed the usual way. I’ve created a script to do most of it for you but here’s the steps. You need to be root.
# run this as root
# download it
curl -o /root/amazon-cloudwatch-agent.deb https://s3.amazonaws.com/amazoncloudwatch-agent\
/debian/amd64/latest/amazon-cloudwatch-agent.deb
# install it
dpkg -i -E /root/amazon-cloudwatch-agent.deb
This will create a user “cwagent” but does nothing to configure anything. You next do this:
# add the cwagent user to the adm group so it can read logs
usermod -aG adm cwagent
# configure the user credentials
cat <<EOF > /home/cwagent/.aws/credentials
[AmazonCloudWatchAgent]
aws_access_key_id = aaaaaaaa
aws_secret_access_key = bbbbbbbb
EOF
# configure the user region
cat <<EOF > /home/cwagent/.aws/config
[AmazonCloudWatchAgent]
output = text
region = us-east-1
EOF
# get your own IAM creds for an IAM user and edit the file
nano /home/cwagent/.aws/credentials
# set ownership
chmod -R cwagent.cwagent /home/cwagent
Moving forward requires two things: you need to install the AWS CLI on your Ubuntu server, and you need to create an IAM user “cwagent.” The AWS docs for the latter are quite clear and easy to follow. When you have the AWS credentials for the new user, you need to add those to the new user on your ubuntu server:
# get your own IAM creds for an IAM user and edit the file
nano /home/cwagent/.aws/credentials
Replace nano with your favorite editor, as desired.
Oh, set the region you want as well. The example uses us-east-1 but choose the right region for you.
Note that you also need to chown the files, since you did all this editing as root.
# set ownership
chmod -R cwagent.cwagent /home/cwagent
Now, this next step is important. If you don’t have collectd installed and you configure to collect metrics (see below) then the agent will just hang and not do anything. Not even send syslogs. I need to test this to ensure I can reproduce it, but it seems like a bug. The logs should at least say something about it but they don’t. More on the agent logs later.
Configuration
This is where I had a lot of head scratching. The documentation from AWS is flat oout WRONG. I have submitted feedback to the page on what I found. Here’s the scoop:
# run the wizard
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
# move the configuration file to the right place and name it properly
mv /opt/aws/amazon-cloudwatch-agent/bin/config.json \ /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
Running the wizard asks you a set of questions and creates a config file. However you need to move it to the etc folder and rename it in order for the agent to be able to find it. The docs on the sections of the file seem pretty accurate, and it’s self explanatory really. I just accepted defaults, except for specifying the actual syslog file.
Again, make sure you installed collectd. Otherwise, the agent just hangs.
Now you are ready. Restart the service.
# restart the service
service amazon-cloudwatch-agent restart
# check status
service amazon-cloudwatch-agent status
# check the logs
tail -f /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log
You want to see something like this:
2020-11-04T14:26:44Z I! Started the statsd service on :8125
2020-11-04T14:26:44Z I! [inputs.socket_listener] Listening on udp://127.0.0.1:25826
2020-11-04T14:26:44Z I! Statsd listener listening on: [::]:8125
2020-11-04T14:26:45Z I! will use file based credentials provider
2020-11-04T14:26:45Z I! [logagent] piping log from syslog/sol1(/var/log/messages) to cloudwatchlogs
Now you know that the agent is sending stuff to CloudWatch. Your next step is to go there and configure dashboards, etc. That’s for another post!
