LibreNMS 1.50.1 packer.io CentOS image high CPU usage

Leave a reply

LibreNMS offers prebuilt OVA VM images using packer.io based on Ubuntu 18 and CentOS 7. If you’ve deployed the CentOS image though, you may be aware of it using an excessive amount of CPU – not an issue present in the Ubuntu image.

A quick look at the system with top will show us php processes with new PIDs spiking the CPU endlessly. Output of ps axf will show the parent process as syslog-ng, whose config has a PHP script defined as a program which it is trying to launch. Turning on debugging in syslog-ng doesn’t return any information on what is happening so when something on an RHEL based system doesn’t make sense, what do you do? Check if SELinux is enforcing and look at your audit logs!

Continue reading →

Understanding iptable’s hashlimit module

16 Replies

I was having trouble understanding the iptables hashlimit module and couldn’t dig up anything that really helped. The man pages are definitely lacking a clear explanation and /proc/net/ipt_hashlimit/ leaves out some information that would clarify things immensely. After some testing I managed to work it all out, so let’s go through it and see if I can help make sense of it for you too.

I’ll try not to assume too much prior knowledge about the module. We’ll be coming at this with the goal of blocking traffic that exceeds a certain amount of packets per second. From the man page:

hashlimit uses hash buckets to express a rate limiting match (like the limit match) for a group of connections using a single iptables rule. Grouping can be done per-hostgroup (source and/or destination address) and/or per-port. It gives you the ability to express “N packets per time quantum per group” or “N bytes per seconds”

Continue reading →

Integrating NextCloud 11 with FreeIPA 4

7 Replies

Nextcloud + FreeIPA

This is a perfect example of banging your head against the wall for a day. Integration information between FreeIPA and just about anything is hard to come by, so I decided to put this short guide together covering it and Nextcloud. This is just a drop in a bucket that desperately needs to be filled. As always, I will endeavour to explain how I figured this out because understanding the how and why is important! Ultimately this investigation resulted in me tracking down a bug in the LDAP app in Nextcloud, and a patch has been merged for the Nextcloud 12 release.

You’ll require a read-only System account in your FreeIPA instance. This is for Nextcloud to bind to for reading users and groups. Basic instructions for creating a user for this purpose are located here on the FreeIPA wiki.

With this in hand and the LDAP / AD Integration app enabled in your Nextcloud instance you can get to configuring. You should be aware of some of your basic configuration settings in FreeIPA, like your basedn which you can find in /etc/ipa/default.conf on your IPA server.

Continue reading →

Measuring and graphing I-frame gaps in videos

2 Replies

I’ve been mired in IPTV related work for a while now, popping my head up occasionally to grab a breath and take a look at some other neglected things in my day-to-day job. It isn’t long before I’m back into it though, and all the time I’m picking up more knowledge about the world of TV.

One thing that was brought to me recently was some failures related to a network PVR system having trouble ingesting encrypted video content. According to the support team for the software, I-frames in the content were too far apart to ingest properly. The person working on the issue at our company was interested in seeing just how far apart they were, so I set to creating a little script to do just that.

It turned out to be relatively simple using ffprobe, which is part of the ffmpeg suite. Using that data, it’s then possible to plot the I-frame gaps with gnuplot.

iframe-distance-plot

I-Frame gaps

Continue reading →

Enabling pam_tty_audit on RHEL, CentOS, or Scientific Linux 6

Leave a reply

aureport output

If you have plans to make use of the audit system in RHEL and its clones, you may decide to use pam_tty_audit. A lot of the top hits on Google will recommend enabling the module in /etc/pam.d/system-auth-ac, but because of changes related to having better support for HIDs like fingerprint scanners, it should actually exist in /etc/pam.d/password-auth-ac, which is included by /etc/pam.d/sshd and others. One other complaint is that it logs passwords, but a patch was implemented in an update to RHEL 6 to address this.

Add the following line to the end of /etc/pam.d/password-auth-ac to enable keystroke logging for all users on a default RHEL, CentOS or Scientific Linux 6 server:

session     required      pam_tty_audit.so enable=*

The raw audit log can then be viewed at /var/log/audit/audit.log (which includes other audit events) or output in a somewhat friendlier format using aureport as root.

aureport --tty -ts today

The audit system caches based on settings in /etc/audit/auditd.conf, so it could take some time before you see audit logs for user activity. Check its man page for further information

The Dog Cam

1 Reply

Dog with camera pack

Pack mounted wireless camera

So you have a dog in your office, work on a residential television service and it’s almost April Fools’. What do you do? Put a camera on it and broadcast it of course. That’s the immediate idea I had anyways, so I spent some time in March figuring out how I could do just that.

Continue reading →

Cleaning up the winsxs folder on Windows 7

Leave a reply

I noticed tonight that I was running very low on space on one of my personal machines whose Windows 7 install lives on a smaller SSD. Space is at a premium there, so I looked for where the bloat was coming from and tracked a large amount of usage to the C:\Windows\winsxs folder. Looking into what its contents were for I found it houses old install information to allow you to back out patches. Sometimes those patches are replaced by newer ones, but the old stuff remains.

Disk Cleanup Utility

The Disk Cleanup utility provided in a regular install can usually take care of this sort of thing. Cleaning that particular folder however requires an optional patch, one that might not be installed onto your system through the automatic update process depending on your settings. By installing KB2852386, running Disk Cleanup as Administrator will allow you to wipe out the cruft, and should free up a good amount of space, and while you won’t be prompted to reboot, you will not see all the space freed up until you have.

The $280 `htop` T-Shirt

Leave a reply

Angelos Frentzos T-Shirt

Angelos Frentzos “Crash” T-Shirt

Having an interest in fashion and an interest in Linux are usually two very separate things for me. I don’t think System Administrators are well known for their style, and I definitely didn’t expect to be writing a post about it here, so imagine my surprise when I came across this particular shirt on antonioli.eu’s latest arrivals. Designed by Angelos Frentzos and priced at a whopping $280 Canadian, it’s quite a bit of money for a tee full of wacky computer speak! Immediately though, I noticed that calling it the “crash print” is a bit of a misnomer, as it looks like pretty typical htop output with some sort of window split. Curious about its origin, I thought I’d look into it.

Continue reading →

Installing ArchLinux AUR packages in a post-package signing world

1 Reply

ArchLinux and I aren’t the best of friends, its rolling release system just doesn’t jive with me. I only really installed it on a couple personal servers for experience with another distribution, so usually the boxes sit there doing their thing until I feel like setting up something new. When I run its first update in six months, something usually goes wrong, and I make my way over to their forums to figure out how to give pacman a kick in the butt to get things sorted.

This time around, I’m installing metasploit to test some equipment against the recent security flaws in UPnP that have been making some waves. A binary package isn’t available in the official repositories so in this case the Arch User Repository (AUR) picks up the slack to automate building from source. I’ve used this in the past with some success, but last year Arch finally implemented package signing. Turns out this complicates installing these personally built packages unless you temporarily disable it. Let’s not do that though, let’s stay consistent with package signing. In a larger environment with your own custom repo, you would definitely want to have this working, and there’s nothing wrong with learning more about your system.

Continue reading →

Aastra phones rebooting after DHCP lease changes

1 Reply

Working at a service provider for hundreds of VoIP customers with thousands of phones means you’re going to encounter problems that make you scratch your head. Every now and then one of those problems gets escalated to me and I’m left figuring out why something is happening. Sometimes that means running packet captures on all sorts of equipment, installing in-line network taps to grab traffic off the wire, or trudging through debug logs. Every now and then I get lucky, figure out the problem before we get started and look like a wizard. More often though, I’m left just as confused as everyone else.

This problem has come to me in a number of iterations, but it’s always intermittent, unexplained phone reboots that appear to have no rhyme or reason. Mid-call or sitting untouched, whatever the usage scenario, the customer was complaining that their phones were rebooting, affecting multiple Aastra models. The last time this was happening with noticeable frequency was several months back, and we thought we nabbed the problem when a provisioning error related to an access control list (ACL) on a switch was fixed. Phones stopped rebooting, the ticket was closed and the customer was happy.

Fast forward to the other day when the problem reared its ugly head again after multiple phones in their office rebooted unexpectedly. The ACL was still in place, so this was something new. What could it be this time?

Continue reading →