Steve Pitcher

Consuming-Journal-Receiver-Data-Easily-with-SQL

Consuming Journal Receiver Data Easily with SQL

A few weeks ago we were tasked with helping a customer log all IBM i user profile changes and deletes for audit and historical purposes.

Of course, you can certainly use the security audit journal for this type of task however you may not want to keep journal receivers on the system forever. The security audit journal receivers can be quite large and frequently generated, depending on the busyness of the server.

Truth be told…I absolutely detest interacting with the audit journal data directly. I always have. The DSPJRN command is clunky and the results look like the journal receiver fell off the ugly tree face-first, making them hard to read and comprehend.

Instead, I generated a couple of tables (user_changes and user_deletes) to collect the information, then retrieved information from the audit journal by way of Run SQL Scripts and two new IBM i services contained in the SYSTOOLS schema:

SYSTOOLS.AUDIT_JOURNAL_CP
SYSTOOLS.AUDIT_JOURNAL_DO

Both of these functions pull basic and entry-specific information from the security audit journal and were released in 7.3 TR 11 and 7.4 TR 5. SYSTOOLS.AUDIT_JOURNAL_CP contains user profile changes while SYSTOOLS.AUDIT_JOURNAL_DO contains object delete events. While the AUDIT_JOURNAL_CP table function gives me everything I need, the AUDIT_JOURNAL_DO will do much more than that. It returns information about EVERY object deletion, so I need to ensure that the object type returned is a user profile object (i.e., *USRPRF).

INSERT INTO
profilelog.user_deletes(entry_timestamp,User_name,
qualified_job_name,object_name)
SELECT entry_timestamp,User_name,qualified_job_name,object_name
FROM TABLE (
SYSTOOLS.AUDIT_JOURNAL_DO(STARTING_TIMESTAMP => CURRENT TIMESTAMP – 1
DAY)
)

WHERE object_type = ‘*USRPRF’;

Now, each day there’s a job run on schedule that will collect those specific journal entries for user profile changes and deletes and insert them into the two logging tables. Could you trigger that type of call by way of a trigger like an exit point program? Absolutely. I simply chose the job scheduler option simply because it’s quick and easy. Depending on the time it runs each day, I could get a couple of duplicate records but that’s not the end of the world…they’re timestamped entries.

From a user profile change perspective, the real value is being to determine not only who changed a user profile but WHAT they changed. Did they have no special authorities before? Was *ALLOBJ special authority added? Was it a password change? Does that password change meet password requirements? AUDIT_JOURNAL_CP will tell you all that and more very quickly…and with plain English …

February 2022 IBM i Security Alert

With the armed conflict in Ukraine developing, we forecast Russian and Belarussian cyber-attacks against Western nations to escalate imminently. This page will be updated as the situation unfolds.

While critical infrastructure customers are likely the most obvious targets, you need to be prepared to defend against cyber-attacks and protect your business no matter your industry. Every organization is at risk.

The following is a quick cheat sheet of proactive advice for your IBM i:

  • Ensure you have a recent and successful full system save readily available
    • On the GO SAVE menu, this happens when you take option 21.
    • If you run Backup and Recovery Media Services, ensure you have a good *SYSTEM backup.
    • If you don’t have a recent full system save then please schedule it ASAP. We can only recover what you save.
  • Download the latest Licensed Internal Code resave for all IBM i releases you are running in case the need arises for a bare-metal recovery
  • Ensure you are auditing security events on your IBM i
    • Run command DSPSECAUD to check your security auditing status
    • Federal law enforcement will want extended logging that QHST and the QSYSOPR message queues do not provide
  • Reduce the amount of read/write-capable file shares
  • Stop sharing critical IBM i directories such as
    • Root (/)
    • /QIBM
    • /QOpenSys
    • /QDLS
  • Do not share critical user data directories if at all possible
  • For all user directory file shares, ensure that proper object security is in place
    • Exclude *public
    • Only allow the users you want to allow access
  • Considerably reduce the number of users with special authorities, especially *ALLOBJ.
  • Ensure you have minimal to zero Network Address Translation rules that directly forward port traffic from your firewall to your IBM i. Do not assume. Have your network team prove this out to you well in advance.
  • Ensure you are up to date on Program Temporary Fixes (PTFs)

More generally speaking:

  • Educate your users that:
    • Foreign cybersecurity attacks are imminent and likely.
    • Any suspicious activity must be reported immediately.
    • Take precautions when opening email attachments or if asked to provide secure information over the phone/Internet

If you suspect you’ve been breached, please contact one of the following federal authorities:

  • United States:
    • Department of Homeland Security
      • Cybersecurity and Infrastructure Security Agency (CISA)
      • United States Secret Service
    • Department of Justice
      • FBI
    • If you notify any single DOJ or DHS entity, the other two are notified on your behalf. There is

A Simple IBM i Penetration Test Lesson – Part 2

A few months ago we talked about two very simple core components of our IBM i penetration test: exploiting firewall NAT rules and default passwords.

I wanted to build upon that by showing you other ways to properly lock down your system by way of showing you how to easily exploit it.

So let’s talk about object authority on user profiles…

Now, most people look at a user profile and think about some of the attributes defined within, such as special authorities, initial menus, password expiry intervals, etc. Those are certainly important, but I want to focus on the object itself.

By default, IBM i user profile objects are given the *public authority of *exclude. That means that users without *allobj special authorities or private authority to the user profile objects have no access to view a user profile or determine anything about the aforementioned attributes. Unfortunately, there are always a few profile objects on a system that have *public authority of *use, *change or *all. And guess what? Those profile objects usually have some sort of special authority. I’m not sure just why that is but my guess is that generally they’re vendor-created profiles. A vendor gets on the system and has the ability to create user profiles and set them up for adopted authority for their software…but they just didn’t really know what they were doing. They end up giving the new user profiles special authorities and then set up the profile object as something other than *public *exclude.

So, when we come into the picture on a white box penetration test to check if we can elevate our authority, the first thing I attempt to do is see what user profiles are not set up as *public *exclude. Why would I do that? It’s a simple vulnerability I want to exploit. It’s an old and simple hack, but a very effective one if you’re not buttoned up. And so you’re aware, there’s a very similar one regarding Job Descriptions when you’re at QSECURITY level 30…but that’s another topic for another day. Quick point…if you’re at QSECURITY 30, you need to be at 40.

Back to the original point. If I have *use, *change or *all private authority to a user profile with some special authorities, then I can simply submit jobs and have them run as that particular user.

For example,

I can do a wrkusrprf *all …

Shut down those /QIBM shares

Last week I posed a question on Twitter and LinkedIn about what would actually be deleted if your IBM i /QIBM share was fully compromised by malware.

What’s the /QIBM share?

Well, it’s something that was released years ago but has since been deemed a security risk. In fact, it was iTech Solutions that reported it and requested it shut down. Since then, four PTFs have been released to do just that; one for each recent OS version:

    • 7.1 — SI76071
    • 7.2 — SI76072
    • 7.3 — SI76073
    • 7.4 — SI76074

Unfortunately, many IBM i customers may not be able to apply PTFs too often so that share may exist on your system. It’s a very simple process to turn it off. Essentially you have to go into IBM Navigator for i, find the file share for /QIBM and click “Stop Sharing.” That’s all the PTF will do as well.

Now, to leave that share out there would not be an ideal thing to do. If someone with *ALLOBJ special authority were to map that drive to their PC or laptop, then inadvertently kicked off some ransomware, then the /QIBM directory would be in big trouble. How much trouble?

Well, I happen to have an IBM i partition we use for such tests. I decided to redeploy some custom malware as I did in the article called The Real Effects of Malware on IBM i against the /QIBM directory on that same (albeit rebuilt) WLECYOTE server. I do this so you don’t have to wonder…you’ll know.

Malware directed at WLECYOTE ran for about 5 minutes, destroying much of /QIBM. What was the result?

Well, licensed programs in *ERROR status were:

  • 5770SS1 options *base, 3, 30, 31, 39, 43
  • 5770DG1
  • 5770JV1
  • 5770WDS options 33, 34, 35, 41, 43, 44, 45

 

  • Host servers mostly destroyed except Telnet
    • Good luck getting Access for Windows or ACS Telnet without Signon
    • DOS Telnet will work! Enjoy!
    • But to be fair, I think I had a Telnet server in use…so your mileage may vary depending on the time of day.
  • Objects missing for licensed programs 5770TC1, 5770TS1, 5770XW1, 5770NAE, 5770PT1, 5733SC1
    • That’s TCPIP, Transform Services, IBM Access Family, Network Authentication Enablement, Performance Tools, SSH
  • License information
  • Digital Certificate Manager…goodbye encryption, even if Telnet works
  • Navigator for i

That’s a good chunk of the system destroyed. It’s functional…sort of.

That’s why you need to update PTFs (to …

January 2022 IBM i Security Alert

Updated as of January 10, 2022 for the affected Products and Versions

On December 2021, CVE-2021-44228 was announced as a critical zero-day vulnerability and detailed the capability of remote code execution on systems using Log4j versions 2.0 through 2.15. This was one of the largest patch updates efforts in history.

IBM is regularly releasing new information on Log4j vulnerabilities and related mitigation recommendations. Other vulnerabilities have been since released, such as CVE-2021-45105, CVE-2021-4104 and CVE-2021-45046, which will also be continued to be investigated with remediation recommendations.

IBM has been publishing subsequent remediation recommendations in their PSIRT blog and security alerts.

Given that Log4j architecture has been continuously investigated over the last month there’s a lot of noise on the subject. People may think that they’re “one and done.” That is not necessarily the case. iTech Solutions will provide more clear insight on risk and remediation as new information becomes available. This blog entry will be a living document, outlining each Log4j CVE and remediation requirements as per IBM for their products.

For customers using versions of Log4j in custom applications, we would encourage you to either upgrade to the latest version of Log4j if possible or investigate different options for logging solutions for those custom applications.

There are a few different ways to determine what versions of Log4J are installed on your system:

  1. Scott Forstie has posted a handy script using IBM i Services located here. Please note that this works for IBM i 7.3 and higher. https://gist.github.com/forstie/9662d4c302f5224c66b7a4c409141a2c
  2. For 7.2 or lower, you can use the following shell script. The script posted the other day did not account for case sensitivity whereas this does:

qsh
cd /
find . -type d \( -path ./QDLS -o -path ./QFileSvr.400 -o -path ./QIMGCLG -o
-path ./QNTC -o -path ./QOPT -o -path ./QSYS.LIB -o -path ./QSYS.LIB \) -prune -o -name ‘*[lL][oO][gG]4[jJ]*’ -print

 

Vulnerability: CVE-2021-45105
Type: DoS
Description: Apache Log4j2 versions 2.0-alpha1 through 2.16.0 (excluding 2.12.3 and 2.3.1) did not protect from uncontrolled recursion from self-referential lookups. This allows an attacker with control over Thread Context Map data to cause a denial of service when a crafted string is interpreted. This issue was fixed in Log4j 2.17.0, 2.12.3, and 2.3.1.
Severity Score: 4.3
Published: 2021-12-18
Affects IBM i: Yes
Affected Products: DB2 Web Query versions 2.2.1 and 2.3.0. IBM WebSphere Application Server versions 8.5 and 9.0.
Remediation:  https://www.ibm.com/support/pages/node/6537454, https://www.ibm.com/support/pages/node/6538148

 

Vulnerability: CVE-2021-45046

Groundhog Day for Malware

Say it with me: IBM i is NOT immune to malware.

A couple of years ago I wrote a piece called The Real Effects of Malware on IBM i. I thought it laid out a pretty fun yet frighteningly serious story of having an argument with a gentleman on Facebook regarding what’s IBM i fact vs fiction regarding malware and how myself and iTech Solutions colleague Nathan Williams proved it out with some homemade malware and hosed a test system in the process. It really just says everything it needs to.

So a few weeks ago I’m on Facebook again having the same argument with other people.

I’ll not besmirch the original poster’s name in this newsletter article. I just want to highlight his content of the conversation so I can add a few formal rebuttals after I’ve had some additional time to ponder. I’ve cleaned it up a little for the benefit of the readers.

The IFS just like a UNIX or Windows file system is susceptible to viruses, the i/OS is NOT.”

Okay, this comment is pretty much false information. First, the IFS is called the Integrated File System because it’s exactly that. It literally contains ALL TEN IBM i file systems! Here they all are for good measure:

Integrated File System

Root File System

QOpenSys

QSYS.LIB

IASP QSYS.LIB

QDLS

QOPT

QFileSvr.400

UDFS

NFS

QNTC

It starts with the Root file system of course.

Every other file system is underneath the root directory. Contained in various places within those file systems is the IBM i operating system. If you expose these file systems through SMB file shares via IBM NetServer, then they are 110% susceptible to malware. See the article above.

No, the IBM OS is NOT susceptible to Malware and PC Viruses…IFS files are of course because they are just PC files anyway, but the architecture of the IBM i and its objects are not going to be attacked by viruses…in my 38 years of IBM midrange including IBM Rochester support, sorry, you are wrong.

Again, there’s a fundamental misunderstanding of what exactly the IFS actually is and what is or isn’t susceptible to malware. And once someone pulls out the years of experience as a reason to accept their argument as gospel then they’ve lost any leg to stand on. It’s a whopping non sequitur. If someone has 50 years in mathematics and …

A Simple IBM i Penetration Test Lesson

iTech Solutions has had IBM i penetration testing as a service for a little while now. While not discussing anything confidential, offering a little friendly advice based on some of the generic and even specific things encountered on a penetration test will certainly be to the benefit of IBM i customers such as yourselves.

First, let’s explain the meat of what iTech does on a pen test.

  1. Perimeter Test – attempts to catalog and breach your systems from outside the firewall.
  2. Black Box Test – attempts to find and breach your IBM i partition(s) from inside the firewall. No knowledge of any system specifics (even the IP address) are given in advance.
  3. White Box Test – attempts to elevate authority of an existing user to get around security rules on the system. This is the equivalent of a rogue user attempting to steal data or do some damage.

For each test, the ground rules are agreed upon by both parties. Some customers want you to spend a little extra time on a particular service (i.e., EDI, web servers) which is certainly doable. Others want nobody in the local IT department to know about it in order to test efficacy, alerting and reaction of an attacker tripping a tripwire.

The minority of shops are watertight. Many are not. Some are far more wide open than they thought. So much so that all three aspects of their penetration tests have taken less than 20 minutes combined. That means breaching the system from outside the firewall with a set of working credentials and then either elevating authority to gain more control or simply by having excess rights with the profile used in the very first place.

How does that happen? It’s usually the combination of two things.

Network Address Translation (NAT) rules on a firewall is the first massive problem. A NAT rule will take a port on a firewall public IP address and map it to a corresponding IP address on a local server. For instance, you may have a public IP address of 142.175.10.xxx and a local address of 10.10.10.50 for your IBM i server with an FTP server running on port 21. External FTP clients would connect to your firewall IP address with an FTP request on port 21 and the firewall would route that request to 10.10.10.50 on port 21. It’s simply a traffic cop. So review your firewall …

Anonymizing the IBM i FTP Banner

Part of securing any system is not waving a big red flag showing the architecture of the server you’re trying to protect.

There are quite a few websites on the Internet used for cataloging the results of port scans. They’ve been on my radar for years because reports of cataloged police body cameras and license plate readers, and not to mention “nanny cams”, made the headlines over a decade ago. This was when the Internet of Things was a fairly hot buzz term. In reality it’s turned into the Internet of Unprotected Things. Back then I got to thinking about IBM i security and what could be exposed by these types of web crawlers.

The biggest culprit is FTP.

Many IBM i shops allow Network Address Translation rules through their firewalls directly to their IBM i. And the FTP server on IBM i is easily identifiable by its banner comprised of the subsystem name (QTCP) and the host name of the system. QTCP is inherently tied to the IBM i operating system which makes it a great term to find IBM i systems in the wild. And as such, here are about 1750 IBM i systems that have been cataloged in the United States alone:

 

While ideally, I would always recommend to front-end your IBM i with an intermediary server to protect it a bit better, what I’m going to show you is how to change the default FTP banner.

That will at least stop waving one big red flag to allow attackers to identify the IBM i operating system.

Do a WRKMSGF MSGF(QTCP/QTCPMSGF) and put a 12 next to it to work with message descriptions. Find MSGID TCP120D. This is your default FTP banner for the system. If you put a 2 next to it you’ll see the following as its message text:

‘220- &1 at &2.’

220 is the default welcome message for any new user connecting to the FTP server. Parameter &1 is the subsystem QTCP and &2 is the host name of the IBM i partition.

What I like to do is leave the 220 and then change the text, removing the parameters that identify the server and instead put something arbitrary in their places.

You can edit it in place, but the shorthand way to do the entire procedure is this:

CHGMSGD MSGID(TCP120D) MSGF(QTCP/QTCPMSGF) MSG(‘220-whatever you want right here’)

So one of our servers …

Changing Your IBM Business Partner

I can’t count how many times I’ve heard someone say the following statement: “I wish I could use iTech Solutions as our IBM Business Partner but…”

 

Many customers think that they’re effectively married to their Business Partner; bound by some kind of unwritten contract so they can’t work with anyone else.

 

This is absolutely false.

 

While there are some rules that IBM Business Partners must adhere to that I can’t go into detail about, for the most part, if someone wants to work with ANY partner, then there is nothing stopping them from doing that. In fact, most of our new customers come the old fashioned way: word of mouth. One customer works with us and tells people in their local IBM i community. And then one good turn deserves another.

 

Other times, some partners go out of business or just stop doing what they should be doing: representing IBM to their customer base. I had one customer a year or so ago who hadn’t heard from their partner in SEVEN years when they installed their POWER7. Do you know who they ended up calling when they needed help? Us. Simply because they knew we were looking to earn their business. And we earned it by dragging their system off the fire one night when their old partner didn’t pick up the phone. I don’t even think we charged them for the hour. All’s fair in love and war I suppose.

 

Your IBM Business Partner Can Be Located Anywhere

Perhaps it’s out of habit, location, or a combination of both. A company has used an IBM Business Partner for 25 years or so and they think that they’re the only one in town. While that may be literally true to some extent, what’s even more true is that you don’t need a local partner anymore. If the COVID pandemic has taught the world anything, it’s that people work from everywhere. That’s why we help customers with services all over the world. While we can only sell IBM hardware in North America, we will gladly work with and recommend other reputable hardware resellers from Belgrade to Bangkok and everywhere in between.

 

Busting Business Partner Myths

And while most IBM Business Partners are made up of honest folks trying to make a living like we are, sometimes you hear just flat-out nonsense that some partners try to get away with telling their